Fundamentals of Computer Vision
Lecture 2
Overview of today’s lecture
• Types of image transformations.
• Image formation
• Point and image processing.
Background Reading
Basic reading:
• Szeliski textbook, Section 3.2
Properties of Vision
• 3D representations are easily constructed • There are many different cues.
• Useful
• to humans (avoid bumping into things; planning a grasp; etc.)
• in computer vision (build models for movies). • Cues include
• Multiple views (motion, stereopsis) • Texture
• Shading
• People draw distinctions between what is seen • “Object recognition”
• How to build programs that can draw useful distinctions based on image properties.
Computer Vision – A Modern Approach Set: Introduction to Vision Slides by D.A. Forsyth
The Physics of Imaging
• How images are formed • Cameras
• What a camera does
• How to tell where the camera was • Light
• How to measure light
• What light does at surfaces
• How the brightness values we see in cameras are determined
• Color
• The underlying mechanisms of color • How to describe it and measure it
Computer Vision – A Modern Approach Set: Introduction to Vision Slides by D.A. Forsyth
Good news: with more than one camera, we can recover 3D!
Example: Stereo Vision
Structure from Motion
We can also infer 3D from only one camera, provided we move it around “enough”.
t-3
t-2
t-4
Tx
pitch
t+2
t-1
roll yaw Ty Tz
t+1
t
Structure from Motion
So how much is “enough” motion? It depends.
More Difficulties
Object appearance changes with respect to viewpoint and lighting.
Image Formation
Slide credit: Derek Hoiem
What is an image?
• A grid (matrix) of intensity values
=
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
20
0
255
255
255
255
255
255
255
255
255
255
75
75
75
255
255
255
255
255
255
255
255
75
95
95
75
255
255
255
255
255
255
255
255
96
127
145
175
255
255
255
255
255
255
255
255
127
145
175
175
175
255
255
255
255
255
255
255
127
145
200
200
175
175
95
255
255
255
255
255
127
145
200
200
175
175
95
47
255
255
255
255
127
145
145
175
127
127
95
47
255
255
255
255
74
127
127
127
95
95
95
47
255
255
255
255
255
74
74
74
74
74
74
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
255
(common to use one byte per value: 0 = black, 255 = white)
Digital images, Is Image a Function ?
• We can think of a (grayscale) image as a function, f, from R2 to R: – f (x,y) gives the intensity at position (x,y)
• A digital image is a discrete (sampled, quantized) version of this function
• Sample the 2D space on a regular grid
• Quantize each sample (round to nearest integer)
• Image thus represented as a matrix of integer values.
f (x, y) x
y
2D
1D
Another Example
grayscale image
What is the range of the image function f?
domain
A (grayscale) image is a 2D function.
What types of image transformations can we do?
Filtering Warping
changes pixel values
changes pixel locations
What types of image transformations can we do?
Filtering
Warping
changes range of image function
changes domain of image function
What types of image filtering can we do?
Point Operation
Neighborhood Operation
point processing
“filtering”
Point processing
Examples of point processing
original darken lower contrast
non-linear lower contrast
invert lighten raise contrast non-linear raise contrast
How would you implement these?
original
darken
lower contrast
non-linear lower contrast
non-linear raise contrast
Examples of point processing
invert
lighten
raise contrast
How would you implement these?
original
darken
lower contrast
non-linear lower contrast
non-linear raise contrast
Examples of point processing
invert
lighten
raise contrast
How would you implement these?
original
darken
lower contrast
non-linear lower contrast
non-linear raise contrast
Examples of point processing
invert
lighten
raise contrast
How would you implement these?
original
darken
lower contrast
non-linear lower contrast
non-linear raise contrast
Examples of point processing
invert
lighten
raise contrast
How would you implement these?
original
darken
lower contrast
non-linear lower contrast
non-linear raise contrast
Examples of point processing
invert
lighten
raise contrast
How would you implement these?
original
darken
lower contrast
non-linear lower contrast
non-linear raise contrast
Examples of point processing
invert
lighten
raise contrast
How would you implement these?
original
darken
lower contrast
non-linear lower contrast
non-linear raise contrast
Examples of point processing
invert
lighten
raise contrast
How would you implement these?
original
darken
lower contrast
non-linear lower contrast
non-linear raise contrast
Examples of point processing
invert
lighten
raise contrast
Image filtering
Image filtering
• Compute a function of the local neighborhood at each pixel in the image
• Function specified by a “filter” or mask saying how to combine values from neighbors.
• Uses of filtering:
• Enhance an image (denoise, resize, etc)
• Extract information (texture, edges, etc) • Detect patterns (template matching)
Slide credit: Kristen Grauman, Adapted from Derek Hoiem
Images in Matlab
• Images represented as a matrix
• Suppose we have a NxM RGB image called “im”
• im(y,x,b)=ypixelsdown,xpixelstorightinthebthchannel • im(1,1,1)=top-leftpixelvalueinR-channel
• im(N,M,3)=bottom-rightpixelinB-channel
• imread(filename) returns a uint8 image (values 0 to 255) • Converttodoubleformat(values0to1)withim2double
row column R
0.92
0.95
0.89
0.96
0.71
0.49
0.86
0.96
0.69
0.79
0.91
0.93
0.89
0.72
0.95
0.81
0.62
0.84
0.67
0.49
0.73
0.94
0.94
0.82
0.51 0.92
0.88 0.95
0.81 0.89
0.60 0.96
0.74 0.71
0.54 0.49
0.56 0.86
0.90 0.96
0.89
0.69 0.79
0.91
0.97
0.89
0.55 0.93
0.94 0.89
0.87 0.72
0.58 0.95
0.58 0.81
0.85 0.62
0.66 0.84
0.67 0.67
0.49
0.49 0.73
0.94
0.62
0.56
0.51 0.94
0.37
0.31
0.42 0.97
0.85
0.75
0.97
0.92
0.93
0.81
0.92
0.95
0.99
0.91
0.56 0.82
0.57 0.51
0.92 0.50
0.88
0.95 0.51
0.81
0.89 0.48
0.60
0.96 0.43
0.74
0.71 0.33
0.54
0.49 0.41
0.56 0.86
0.90 0.96
0.89 0.69
0.79
0.91
0.46 0.89
0.37 0.55
0.93 0.60
0.94
0.89 0.39
0.87
0.72 0.37
0.58
0.95 0.42
0.58
0.81 0.61
0.85
0.62 0.78
0.66 0.84
0.67 0.67
0.49 0.49
0.73
0.94
0.57 0.62
0.91 0.56
0.80 0.51
0.94 0.58
0.56
0.82 0.73
0.57
0.51 0.88
0.50
0.88 0.77
0.51
0.81 0.69
0.48
0.60 0.78
0.43 0.74
0.33 0.54
0.41 0.56
0.90
0.89
0.41 0.37
0.87 0.31
0.88 0.42
0.97 0.50
0.46
0.89 0.92
0.37
0.55 0.90
0.60
0.94 0.73
0.39
0.87 0.79
0.37
0.58 0.77
0.42 0.58
0.61 0.85
0.78 0.66
0.67
0.49
0.49 0.85
0.90 0.75
0.89 0.57
0.62 0.61
0.91
0.56 0.91
0.80
0.51 0.94
0.58
0.56 0.71
0.73
0.57 0.73
0.88
0.50 0.89
0.77 0.51
0.69 0.48
0.78 0.43
0.33
0.41
0.91 0.97
0.97 0.92
0.79 0.41
0.37 0.45
0.87
0.31 0.49
0.88
0.42 0.82
0.50
0.46 0.90
0.92
0.37 0.93
0.90
0.60 0.99
0.73 0.39
0.79 0.37
0.77 0.42
0.61
0.78
0.92 0.93
0.95 0.81
0.85 0.49
0.85 0.33
0.90
0.75 0.74
0.89
0.57 0.93
0.61
0.91 0.99
0.91
0.80 0.97
0.94
0.58 0.93
0.71 0.73
0.73 0.88
0.89 0.77
0.69
0.78
0.92
0.95
0.91 0.97
0.97 0.92
0.79 0.41
0.45 0.87
0.49 0.88
0.82 0.50
0.90 0.92
0.93 0.90
0.99 0.73
0.79
0.77
0.99
0.91
0.92 0.93
0.95 0.81
0.85 0.49
0.33 0.90
0.74 0.89
0.93 0.61
0.99 0.91
0.97 0.94
0.93 0.71
0.73
0.89
0.92
0.95
0.91
0.97
0.79
0.45
0.49
0.82
0.90
0.93
0.99
0.99
0.91
0.92
0.95
0.85
0.33
0.74
0.93
0.99
0.97
0.93
G
B
Slide credit: Derek Hoiem
Common Types of Noise
• Salt and pepper noise: random occurrences of black and white pixels
• Impulse noise: random occurrences of white pixels
• Gaussian noise: variations in intensity drawn from a Gaussian normal distribution
Slide credit: Steve Seitz
Motivation: Noise Reduction
Slide credit: Adapted from Kristen Grauman
Motivation: noise reduction
• Evenmultipleimagesofthesamestaticscenewillnotbe identical.
• Howcouldwereducethenoise,i.e.,giveanestimateofthe true intensities?
Slide credit: Kristen Grauman
>> noise = randn(size(im)).*sigma; >> output = im + noise;
What is impact of the sigma?
Slide credit: Kristen Grauman Figure from Martial Hebert
Gaussian noise
Linear shift-invariant image filtering
Linear shift-invariant image filtering
• Replace each pixel by a linear combination of its neighbors (and possibly itself).
• The combination is determined by the filter’s kernel.
• The same kernel is shifted to all pixel locations so that all pixels use the same
linear combination of their neighbors.
• Assumptions:
Expect pixels to be like their neighbors
Expect noise processes to be independent from pixel to pixel
First attempt at a solution
• Let’s replace each pixel with an average of all the values in its neighborhood
• Moving average in 1D:
Slide credit: S. Marschner
Weighted Moving Average
• Can add weights to our moving average • Weights [1,1,1,1,1] /5
Slide credit: S. Marschner
Weighted Moving Average
• Non-uniform weights [1, 4, 6, 4, 1] / 16
Slide credit: S. Marschner
How about 2D Filtering? Example: the box filter
• Also known as the 2D rect filter
• Also known as the square mean filter
kernel
• replaces pixel with local average
• has smoothing (blurring) effect
1
1
1
1
1
1
1
1
1
Let’s run the box filter
image
output
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
kernel
note that we assume that the kernel coordinates are centered
output
filter
image (signal)
1
1
1
1
1
1
1
1
1
Let’s run the box filter
image output
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image
output
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
kernel
shift-invariant: as the pixel shifts, so does the kernel
1
1
1
1
1
1
1
1
1
output
filter
image (signal)
Let’s run the box filter
image output
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10
20
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10
20
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10
20
30
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10
20
30
30
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10
20
30
30
30
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10
20
30
30
30
20
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10
20
30
30
30
20
10
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
10
20
30
30
30
20
10
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
10
20
30
30
30
20
10
0
20
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
10
20
30
30
30
20
10
0
20
40
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
10
20
30
30
30
20
10
0
20
40
60
60
60
40
20
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
10
20
30
30
30
20
10
0
20
40
60
60
60
40
20
0
30
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
10
20
30
30
30
20
10
0
20
40
60
60
60
40
20
0
30
50
80
80
90
60
30
0
30
50
80
80
90
60
30
0
20
30
50
50
60
40
20
0
10
20
30
30
30
20
10
10
10
10
10
0
0
0
0
10
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Let’s run the box filter
image output
0
10
20
30
30
30
20
10
0
20
40
60
60
60
40
20
0
30
50
80
80
90
60
30
0
30
50
80
80
90
60
30
0
20
30
50
50
60
40
20
0
10
20
30
30
30
20
10
10
10
10
10
0
0
0
0
10
10
10
10
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
… and the result is
image output
0
10
20
30
30
30
20
10
0
20
40
60
60
60
40
20
0
30
50
80
80
90
60
30
0
30
50
80
80
90
60
30
0
20
30
50
50
60
40
20
0
10
20
30
30
30
20
10
10
10
10
10
0
0
0
0
10
10
10
10
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
kernel
1
1
1
1
1
1
1
1
1
output filter image (signal)
Some more realistic examples
Some more realistic examples
Convolution
Convolution for 1D continuous signals
Definition of filtering as convolution:
notice the flip filtered signal filter input signal
Convolution for 1D continuous signals
Definition of filtering as convolution:
notice the flip
filtered signal filter input signal
Consider the box filter example:
1D continuous box filter
filtering output is a blurred version of g