The University of Sydney Page 70
Extension
The University of Sydney Page 71
Zeiler and Fergus, ECCV 2014
– A non-parametric view of invariance, showing which patterns from the
training set activate the feature map
– Using deconvolution to project the feature activations back to the input
pixel space
Visualize features
Input image
Feedforward (classification)
Pick a single intermediate neuron, e.g. one
value in 128 x 13 x 13 conv5 feature map
Deconvolution (visualization)
Compute gradient of neuron value with respect to image pixels
The University of Sydney Page 72
Deconvolutional Method
(Zeiler and Fergus, 2014)
– For one convolutional layer.
F
eed
-forw
ard
B
ack
w
ard
The University of Sydney Page 73
Deconvolutional Method
– Conv and Deconv
– Use learned filters {-} to convolve forward signal, but use transposed
versions {-⊺} for backward signal.
(Zeiler and Fergus, 2014)
)!,# )!,$
)#,! )#,# )#,$
)$,! )$,# )$,$
)!,!
*!,!% *!,#% *!,$% *!,&%
*#,!% *#,#% *#,$% *#,&%
*$,!% *$,#% *$,$% *$,&%
*&,!% *&,#% *&,$% *&,&%
*!,!%’! *!,#%’!
*#,!%’! *#,#%’!
+⊺×(*!,!%)!, ⋯ , *#,#%)!) ⊺= (*!,!% , ⋯ , *&,&% ) ⊺
+%)!×(*!,!%)!, ⋯ , *&,&%)!) ⊺= (*!,!% , ⋯ , *#,#% ) ⊺
The University of Sydney Page 74
Deconvolutional Method
– Activation function
– Both feedforward and backward processes employ ReLU as activation
functions to ensure feature maps are always positive.
(Zeiler and Fergus, 2014)
EFGH 8 = max(0, 8)
The University of Sydney Page 75
(Zeiler and Fergus, 2014)
Deconvolutional Method
– Pooling and Unpooling
– Despite the non-invertibility of max pooling, an approximate inverse
could be obtained by recording the locations of the maxima within each
pooling region in a set of switch variables.
The University of Sydney Page 76
Visualize features
– Instead of showing the single strongest activation for a given
feature map, the article shows the top 9 activations layer 2-5.
– Alongside these visualizations are the corresponding image
patches.
(Zeiler and Fergus, 2014)
The University of Sydney Page 77
Visualize features
(Zeiler and Fergus, 2014)
– Layer 2 responds to corners and other edge/color conjunctions.
– Layer 3 has more complex invariances, capturing similar textures (e.g.
mesh patterns).
– Layer 4 shows significant variation, but is more class-specific.
– Layer 5 shows entire objects with significant pose variation.
The University of Sydney Page 78
– Occlude images at different locations and visualize feature
activations and classifier confidence
Occlusion Experiments
(Zeiler and Fergus, 2014)
(a) Input Image
(b) Layer 5,
strongest feature map
(c) Classier, probability
of correct class
The University of Sydney Page 79
Variations of CNNs
The University of Sydney Page 80
– Transposed Convolution
– Also named “Deconvolution”, “up-convolution” …
– Usually, the output is an image
– Different from the above ‘Deconvolutional method’ for
visualizing CNNs learned features
Transposed Convolution
Input images 3 Transformed 1(3)
Convolution Transposed Convolution
The University of Sydney Page 81
– Applications
– Generative model, encoder-decoder, image editing, image
transformation …
Transposed Convolution
(Chaoyue et al, 2017)
The University of Sydney Page 82
– Compare to aforementioned convolution operation, can
we go the other way around, i.e., map from a 4-dim space
to a 16-dim space, while keeping the connectivity pattern
of the original convolution.
Transposed Convolution
Convolution operation
%!,# %!,$
%#,! %#,# %#,$
%$,! %$,# %$,$
%!,!
&!,!%&! &!,#%&! &!,$%&! &!,’%&!
&#,!%&! &#,#%&! &#,$%&! &#,’%&!
&$,!%&! &$,#%&! &$,$%&! &$,’%&!
&’,!%&! &’,#%&! &’,$%&! &’,’%&!
&!,!% &!,#%
&#,!% &#,#%
3,-% 4,-% (6,-%) 3,
The University of Sydney Page 83
– Compare to aforementioned convolution operation, can
we go the other way around, i.e., map from a 4-dim space
to a 16-dim space, while keeping the connectivity pattern
of the original convolution.
Transposed Convolution
&%,# &%,$
&#,% &#,# &#,$
&$,% &$,# &$,$
&%,%
8%,%
, 8%,#
, 8%,$
, 8%,.
,
8#,%
, 8#,#
, 8#,$
, 8#,.
,
8$,%
, 8$,#
, 8$,$
, 8$,.
,
8.,%
, 8.,#
, 8.,$
, 8.,.
,
8%,%
,-% 8%,#
,-%
8#,%
,-% 8#,#
,-%
Transposed Convolution
The University of Sydney Page 84
Transposed Convolution
1 0 −1
−1 0 2
0 0 1
0 −1
−1 0 2
0 0 1
1
1 3
-2 -1
1 0 −1
−1 0 2
0 0 1
The University of Sydney Page 85
Transposed Convolution
1 0 −1
−1 0 2
0 0 1
0 −1
−1 0 2
0 0 1
1
1 3
-2 -1
1 0 −1
−1 0 2
0 0 1
3 0 −3
−3 0 6
0 0 3
−3
6
3
1 3 −1
−1 −3 2
0 0 1
The University of Sydney Page 86
Transposed Convolution
1 0 −1
−1 0 2
0 0 1
0 −1
−1 0 2
0 0 1
1
1 3
-2 -1
3 0 −3
−3 0 6
0 0 3
−2 0 2
2 0 −4
0 0 −2
−3
6
3
1 3 −1
−1 −3 2
0 0 1
−3
6
3
1 3 −1
−3 −3 4
2 0 −3
0 0 −1
The University of Sydney Page 87
Transposed Convolution
1 0 −1
−1 0 2
0 0 1
0 −1
−1 0 2
0 0 1
1
1 3
-2 -1
3 0 −3
−3 0 6
0 0 3
−2 0 2
2 0 −4
0 0 −2
−1 0 1
1 0 −2
0 0 −1
−3
6
3
1 3 −1
−3 −3 4
2 0 −3
0 0 −1
−3
7
1
1 3 −1
−3 −4 4
2 1 −3
0 0 −1 2
The University of Sydney Page 88
Stride = 1
Transposed Convolution
– Stride
0 −1
−1 0 2
0 0 1
1
1 3
-2 -1
1 0 −1
−1 0 2
0 0 1
−3
6
3
1 3 −1
−1 −3 2
0 0 1
”
The University of Sydney Page 89
Stride = 2
Transposed Convolution
– Stride
0 −1
−1 0 2
0 0 1
1
1 3
-2 -1
1 0 −1
−1 0 2
0 0 1
”
0
0
0
1 0 2
−1 0 −1
0 0 1
−3
6
3
The University of Sydney Page 90
Transposed Convolution
– Cropping
0 −1
−1 0 2
0 0 1
1
1 3
-2 -1
0
0
0
1 0 2
−1 0 −1
−2 0 2
−3
6
4
0
0
1 0 −4
0 0 −1
4
2
The University of Sydney Page 91
Crop = 1
Transposed Convolution
– Cropping
0 −1
−1 0 2
0 0 1
1
1 3
-2 -1
‘
0
0
0
1 0 2
−1 0 −1
−2 0 2
−3
6
4
0
0
1 0 −4
0 0 −1
4
2
−1 0
0 2 0
0 −4 0
0
The University of Sydney Page 92
Transposed Convolution
– Output Size
Output Size = ; − 1 < + = − 2>
N
# Outpu
t Siz
e
”
C
C
The University of Sydney Page 93
%!,# %!,$
%#,! %#,# %#,$
%$,! %$,# %$,$
%!,!
Convolution as a matrix operation
&!,!%&! &!,#%&! &!,$%&! &!,’%&!
&#,!%&! &#,#%&! &#,$%&! &#,’%&!
&$,!%&! &$,#%&! &$,$%&! &$,’%&!
&’,!%&! &’,#%&! &’,$%&! &’,’%&!
&!,!% &!,#%
&#,!% &#,#%
-()”×(%”,”
()”, ⋯ , %+,+
()”) ⊺= (%”,”
( , ⋯ , %#,#
( ) ⊺
– Recall the matrix operation of original convolution
(!”# =
*#,# *#,% *#,& 0 *%,# *%,% *%,& 0
0 *#,# *#,% *#,& 0 *%,# *%,% *%,&
*&,# *&,% *&,& 0 0 0 0 0
0 *&,# *&,% *&,& 0 0 0 0
0 0 0 0 *#,# *#,% *#,& 0
0 0 0 0 0 *#,# *#,% *#,&
*%,# *%,% *%,& 0 *&,# *&,% *&,& 0
0 *%,# *%,% *%,& 0 *&,# *&,% *&,&
3,-% 4,-% (6,-%) 3,
The University of Sydney Page 94
Transposed Convolution as a matrix operation
4 =
6!,! 6!,# 6!,$ 0 6#,! 6#,# 6#,$ 0
0 6!,! 6!,# 6!,$ 0 6#,! 6#,# 6#,$
6$,! 6$,# 6$,$ 0 0 0 0 0
0 6$,! 6$,# 6$,$ 0 0 0 0
0 0 0 0 6!,! 6!,# 6!,$ 0
0 0 0 0 0 6!,! 6!,# 6!,$
6#,! 6#,# 6#,$ 0 6$,! 6$,# 6$,$ 0
0 6#,! 6#,# 6#,$ 0 6$,! 6$,# 6$,$
– Given the same matrix ?, transposed convolution can be represented as:
Forward pass of transposed convolution
-⊺×(%”,”
()”, ⋯ , %#,#
()”) ⊺= (%”,”
( , ⋯ , %+,+
( ) ⊺
&%,# &%,$
&#,% &#,# &#,$
&$,% &$,# &$,$
&%,%
8%,%
, 8%,#
, 8%,$
, 8%,.
,
8#,%
, 8#,#
, 8#,$
, 8#,.
,
8$,%
, 8$,#
, 8$,$
, 8$,.
,
8.,%
, 8.,#
, 8.,$
, 8.,.
,
8%,%
,-% 8%,#
,-%
8#,%
,-% 8#,#
,-%
The University of Sydney Page 95
Max pooling & Up-pooling
Max pooling Up-pooling
– Pooling ops: max(:) – Fill numbers to pre −define
d position, others fill zeros.
-1 4 1 2
0 1 3 -2
1 5 -2 6
3 -1 -2 -2
4 3
5 6
Input
O
utput
4 0 3 0
0 0 0 0
5 0 6 0
0 0 0 0
4 3
5 6
O
utput
Input
The University of Sydney Page 96
– From features to image
– Kernel: 1 → K×L
– Cropping
– ⋯ → Filtering → Activate
function → Up−pooling → ⋯
Transposed Convolution
C
onvolution
T
ransposed
C
onvolution
Convolution Transposed convolution
– From image to features
– Kernel: K×L → 1
– Padding
– ⋯ → Filtering →Activate
function → Pooling → ⋯
https://github.com/vdumoulin/conv_arithmetic
The University of Sydney Page 97
Thank you!