4/8/2021
CSE 473/573
Introduction to Computer Vision and Image Processing
‘-
STEREO VISION
Slides Credits: Jim Hays++
‘-
1
4/8/2021
‘-
Slides by
Kristen Grauman
Multiple views
stereo vision structure from motion optical flow
‘-
Lowe
Hartley and Zisserman
2
4/8/2021
Why multiple views?
• Structure and depth are inherently ambiguous from single views.
‘-
Images from Lana Lazebnik
‘-
3
4/8/2021
Is depth ambiguous from a single view? • Draw a simple diagram to support your answer
‘-
Exercise 3 minutes
4/5/2021
7
Why multiple views?
• Structure and depth are inherently ambiguous from
single views.
P1
P2
P1’=P2’
‘-
Optical center
4
4/8/2021
What cues help us to perceive 3D shape and depth?
• Shading
• Focus/Defocus
• Texture ‘- • Perspective
• Motion
• Occlusion
Exercise 3 minutes
Shading
‘-
[Figure from Prados & Faugeras 2006]
5
4/8/2021
Focus/defocus
‘-
Images from same point of view, different camera parameters
3d shape / depth estimates
[figs from H. Jin and P. Favaro, 2002]
Texture
‘-
[From A.M. Loh. The recovery of 3-D structure using visual texture patterns. PhD thesis]
6
4/8/2021
Perspective effects
‘-
Image credit: S. Seitz
Motion
‘-
Figures from L. Zhang http://www.brainconnection.com/teasers/?main=illusion/motion-shape
7
4/8/2021
Occlusion
‘-
Rene Magritt’e famous painting Le Blanc- Seing (literal translation: “The Blank Signature”) roughly translates as “free hand“. 1965
‘-
If stereo were critical for depth perception, navigation, recognition, etc., then this would be a problem
8
4/8/2021
M• Sutlrtuic-vtuirew: Ggivenopmroejetcrtiyonps roof tbhelesmamse 3D point in two or more images, compute the 3D coordinates of that
point
?
‘-
Camera 1
R1,t1
Camera 2
R2,t2
Camera 3
R3,t3
Slide credit: Noah Snavely
• Stereo correspondence: Given a point in one of the images, where could its corresponding points be in the other images?
‘-
Camera 1 Camera 2
Camera 3
R3,t3
R1,t1 R ,t 2 2
Slide credit: Noah Snavely
9
4/8/2021
• Motion: Given a set of corresponding points in two or more images, compute the camera parameters
‘-
Camera 1 ? R1,t1
Camera 3
Camera 2
? R ,t 33
R2,t2 ?
Slide credit: Noah Snavely
Estimating scene shape
• “Shape from X”: Shading, Texture, Focus, Motion…
• Stereo:
• shape from “motion” between two views
• infer 3d shape of scene from two (multiple) images from different viewpoints
‘-
Main idea:
scene point
image plane optical center
10
4/8/2021
Human eye
Rough analogy with human visual system:
Pupil/Iris – control
amount of light
‘- passing through lens
Retina – contains sensor cells, where image is formed
Fovea – highest concentration of cones
Fig from Shapiro and Stockman
Human stereopsis: disparity
Human eyes fixate on point in space – rotate so that corresponding images form in centers of fovea.
‘-
11
4/8/2021
Disparity occurs when eyes fixate on one object; others‘-appear at different visual angles
‘-
Disparity: d = r-l = D-F. Forsyth & Ponce
12
4/8/2021
Random dot stereograms
‘-
Random dot stereograms
‘-
From Palmer, “Vision Science”, MIT Press
13
4/8/2021
Random dot stereograms
• When viewed monocularly, they appear random; when
viewed stereoscopically, see 3D structure.
• Human binocular fusion not directly associated with the physical retinas; must involve the central nervous
system (V2, for instance).
‘-
• Imaginary “cyclopean retina” that combines the left and right image stimuli as a single unit
• High level scene understanding not required for Stereo
• But, high level scene understanding is arguably better than stereo
Stereo photography and stereo viewers
Invented by Sir Charles Wheatstone, 1838
Take two pictures of the same subject from two slightly different viewpoints and display so that each eye sees only one of the images.
‘-
Image from fisher-price.com
14
4/8/2021
‘-
http://www.johnsonshawmuseum.org
‘-
http://www.johnsonshawmuseum.org
15
4/8/2021
‘-
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923
‘-
http://www.well.com/~jimg/stereo/stereo_list.html
16
4/8/2021
Autostereograms
Exploit disparity as depth cue using single
‘-
image.
(Single image random dot stereogram, Single image stereogram)
Images from magiceye.com
Exercise Until you solve it
Autostereograms
‘-
Images from magiceye.com
17
4/8/2021
Estimating depth with stereo
• Stereo: shape from “motion” between two views • We’ll need to consider:
• Info on camera pose (“calibration”) ‘- • Image point correspondences
scene point
image plane
optical center
Stereo vision
‘-
Two cameras, simultaneous Single moving camera and views static scene
18
4/8/2021
Recall Camera parameters
Camera frame 2
Extrinsic parameters:
Camera frame 1Camera frame 2
Intrinsic parameters:
Image coordinates relative to
Camera frame 1
Intrinsic params: focal length, pixel sizes (mm), image center point, radial distortion parameters
We’ll assume for now that these parameters are given and fixed.
‘-
• •
camera Pixel coordinates Extrinsic params: rotation matrix and translation vector
Geometry for a simple stereo system • First, assuming parallel optical axes, known camera
parameters (i.e., calibrated cameras):
‘-
19
4/8/2021
World point
image point (left)
Focal length
optical center (left)
Depth of p
‘- (right)
optical center (right)
image point
baseline
Geometry for a simple stereo system • Assume parallel optical axes, known camera parameters
(i.e., calibrated cameras). What is expression for Z? Similar triangles (pl, P, pr) and (Ol, P, Or):
Txx T lr
Zf Z
Zff T
xl – xr
disparity
‘-
20
4/8/2021
Depth from disparity
image I(x,y) Disparity map D(x,y)
‘-
image I ́(x ́,y ́)
(x ́,y ́)=(x+D(x,y), y)
So if we could find the corresponding points in two images, we could estimate relative depth…
Where do we need to search?
‘-
21
General case, with calibrated cameras • The two cameras need not have parallel optical axes.
‘-
4/8/2021
Vs.
Grauman
Stereo correspondence constraints
‘-
• Given p in left image, where can corresponding point p’ be?
22
4/8/2021
Stereo correspondence constraints
‘-
• Given p in left image, where can corresponding point p’ be?
Stereo correspondence constraints
‘-
23
4/8/2021
Stereo correspondence constraints
•Geometry of two views allows us to constrain where the corresponding pixel for some image point in the first view must occur in the second view.
epipolar line
epipolar plane
epipolar line
‘-
Epipolar constraint: Why is this useful?
• Reduces correspondence problem to 1D search along conjugate epipolar lines
Adapted from Steve Seitz
Epipolar geometry
‘-
• Epipolar Plane • Baseline
• Epipoles • Epipolar Lines Adapted from M. Pollefeys, UNC
24
4/8/2021
Epipolar geometry: terms
• Baseline: line joining the camera centers
• Epipole: point of intersection of baseline with the image plane
• Epipolar plane: plane containing baseline and world point ‘-
• Epipolar line: intersection of epipolar plane with the image plane
• All epipolar lines intersect at the epipole
• An epipolar plane intersects the left and right image planes in epipolar lines
Grauman
Epipolar constraint
‘-
• Potential matches for p have to lie on the corresponding epipolar line l’.
• Potential matches for p’ have to lie on the corresponding epipolar line l.
http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html
Source: M. Pollefeys
25
4/8/2021
Next Lecture • Stereo II
• Camera Calibration
‘-
4/5/2021
53
26