Stereo Vision
some slides borrowed or adapted from: •
DTU Electrical Engineering
Copyright By PowCoder代写 加微信 powcoder
• What is Stereo Vision?
• Stereo/Epipolar Geometry
• Rectified Stereo Case
• Depth from Stereo Matches • Correspondence Problem
• Dense vs Sparse Correspondence • Local vs Global Correspondence
• (Dis)-Similarity Measures
DTU Electrical Engineering
• What is Stereo Vision?
• Stereo/Epipolar Geometry
• Rectified Stereo Case
• Depth from Stereo Matches • Correspondence Problem
• Dense vs Sparse Correspondence • Local vs Global Correspondence
• (Dis)-Similarity Measures
DTU Electrical Engineering
What is Stereo Vision?
DTU Electrical Engineering
What is Stereo Vision?
• 3D cinema
• 3D television •…
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Torralba
What is Stereo Vision?
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Torralba
What is Stereo Vision?
DTU Electrical Engineering
Stereo Vision Computation
DTU Electrical Engineering
• What is Stereo Vision?
• Stereo/Epipolar Geometry
• Rectified Stereo Case
• Depth from Stereo Matches • Correspondence Problem
• Dense vs Sparse Correspondence • Local vs Global Correspondence
• (Dis)-Similarity Measures
DTU Electrical Engineering
Camera Model
• Pinhole model
DTU Electrical Engineering
Camera Model
• Pinhole model
DTU Electrical Engineering
Epipolar Geometry
image plane 1
scene point
image plane 2
optical center 1
optical center 2
DTU Electrical Engineering
Epipolar Geometry
If we see a point in camera 1, are there any constraints on where we will find it on camera 2?
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Geometry
If we see a point in camera 1, are there any constraints on where we will find it on camera 2?
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Geometry
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Geometry
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Geometry
Baseline: the line connecting the two camera centers
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Geometry
Baseline: the line connecting the two camera centers Epipole: point of intersection of baseline with the image plane
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Geometry
epipolar plane p
Baseline: the line connecting the two camera centers Epipole: point of intersection of baseline with the image plane
Epipolar plane: the plane that contains the two camera centers and a 3D point in the world
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Geometry
epipolar line
epipolar line
Baseline: the line connecting the two camera centers Epipole: point of intersection of baseline with the image plane
Epipolar plane: the plane that contains the two camera centers and a 3D point in the world Epipolar line: intersection of the epipolar plane with each image plane
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Geometry
epipolar line
• We can search for matches across epipolar lines.
• Search space for correspondences reduces to a 1D problem!
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Geometry
epipolar line
• We can search for matches across epipolar lines
• Search space for correspondences reduces to a 1D problem!
• All epipolar lines intersect at the epipoles
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Geometry
• We can search for matches across epipolar lines
• Search space for correspondences reduces to a 1D problem!
• All epipolar lines intersect at the epipoles
• Where are the epipoles in this case?
DTU Electrical Engineering
Figure from Hartley & Zisserman
• What is Stereo Vision?
• Stereo/Epipolar Geometry
• Rectified Stereo Case
• Depth from Stereo Matches • Correspondence Problem
• Dense vs Sparse Correspondence • Local vs Global Correspondence
• (Dis)-Similarity Measures
DTU Electrical Engineering
Rectified Stereo – a simpler case
image plane 1
optical center 1
scene point
image plane 2
optical center 2
DTU Electrical Engineering
Rectified Stereo – a simpler case
scene point
image plane 1
image plane 2
DTU Electrical Engineering
Rectified Stereo – a simpler case
• Rectification:
– The initial images are reprojected on a common plane that is parallel to the baseline B joining the optical centers of the initial images.
– Epipolar lines become parallel (and under certain conditions they become also horizontal)
Rectification
DTU Electrical Engineering
Rectified Stereo
DTU Electrical Engineering
Rectified Stereo
• “All epipolar lines intersect at the epipoles”
• Where are the epipoles in this case?
DTU Electrical Engineering
• What is Stereo Vision?
• Stereo/Epipolar Geometry
• Rectified Stereo Case
• Depth from Stereo Matches • Correspondence Problem
• Dense vs Sparse Correspondence • Local vs Global Correspondence
• (Dis)-Similarity Measures
DTU Electrical Engineering
Depth from Stereo Matches
• Let us assume (for now!!) that:
• we can check the points along the epipolar line and
• we can find the point (on the right image) that is most similar to our reference point (in the left image), i.e. we can solve the correspondence problem!
DTU Electrical Engineering
Depth from Stereo Matches
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Torralba
Depth from Stereo Matches
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Torralba
Depth from Stereo Matches
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Torralba
Depth from Stereo Matches
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Torralba
Depth from Stereo Matches
B Similar triangles
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Torralba
Depth from Stereo Matches
B Similar triangles
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Torralba
Depth from Stereo Matches
Similar triangles:
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Torralba
Depth from Stereo Matches
Similar triangles:
B-XL+XR = B xL xR Z-fZ
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Torralba
Depth from Stereo Matches
Similar triangles:
B-XL+XR = B xL xR Z-fZ
Solving for Z:
Z=f Disparity
DTU Electrical Engineering
Adapted from W. T. Freeman, P. Isola, A. Torralba
Depth from Stereo Matches
DTU Electrical Engineering
Depth from Stereo Matches
Reference Image
Disparity Map
DTU Electrical Engineering
• What is Stereo Vision?
• Stereo/Epipolar Geometry
• Rectified Stereo Case
• Depth from Stereo Matches • Correspondence Problem
• Dense vs Sparse Correspondence • Local vs Global Correspondence
• (Dis)-Similarity Measures
DTU Electrical Engineering
Correspondence Problem
• We have assumed (up to now!!) that:
• we can check the points along the epipolar line and
• we can find the point (on the right image) that is most similar to our reference point (in the left image), i.e. we can solve the correspondence problem!
• How can we indeed match corresponding pixels between the two stereo images?
DTU Electrical Engineering
Correspondence Problem
• Beyond the hard constraint of epipolar geometry, there are “soft” constraints to help identify corresponding points
– Similarity
– Uniqueness
– Ordering
– Disparity gradient is limited
DTU Electrical Engineering
Correspondence Problem
• To find matches in the image pair, we will assume
– Most scene points visible from both views
– Image regions for the matches are similar in appearance
DTU Electrical Engineering
Correspondence Problem
• It depends!
– Do we need dense or sparse stereo matching?
Stereo Vision
Sparse output
Dense output
•Local Methods (Area-based) •Global Methods (Energy-based) •Other Methods
DTU Electrical Engineering
Sparse Stereo Correspondence
• Extract features (e.g. SIFT, SURF, Harris,…) and match them!
– Pros? – Cons?
DTU Electrical Engineering
Dense Stereo Correspondence : Local Methods
• Try to find correspondences for all the pixels of the reference image. • For each epipolar line
– For each pixel in the left image
• Compare with every pixel on same epipolar line in right image
• Choose the pixel that maximizes a similarity metric (or minimizes a dissimilarity metric!).
DTU Electrical Engineering
Dense Stereo Correspondence
• Try to find correspondences for all the pixels of the reference image. • For each epipolar line
– For each pixel in the left image
• Compare with every pixel on same epipolar line in right image
• Choose the pixel that maximizes a similarity metric (or minimizes a dissimilarity metric!).
• Improvement: don’t match individual pixels, but rather match windows!
DTU Electrical Engineering
Stereo Correspondence Metrics
• Sum of Absolute Differences (SAD)
• Sum of Squared Differences (SSD)
• Normalized Cross-Correlation
• …many many more!!!
DTU Electrical Engineering
Stereo Correspondence Metrics: SSD
DTU Electrical Engineering
Stereo Correspondence Metrics: SSD on various windows
• Small vs Big windows
• What are their Pros and Cons?
W = 3 W = 20
DTU Electrical Engineering
Figure from Li Zhang
Stereo Correspondence Metrics: Good/Bad areas
• In this stereo image pair:
– what would be good areas to match?
– where would you expect to face problems and why?
DTU Electrical Engineering
Figure from Hartley & Zisserman
Global Stereo Correspondence
• Up to this point, the disparity of each pixel was determined only by the information of the pixel itself and its neighborhood.
– Thus, those methods are called ”local” or “area-based” methods. • Example: Result of a local SSD algorithm with W=21:
DTU Electrical Engineering
Global Stereo Correspondence
• Up to this point, the disparity of each pixel was determined only by the information of the pixel itself and its neighborhood.
– Thus, those methods are called ”local” or “area-based” methods.
• Global methods find better solutions in expense of more computations
– Optimize jointly the disparity values of all the pixels of each scanline (e.g. Dynamic Programming)
DTU Electrical Engineering
Global Stereo Correspondence
• Up to this point, the disparity of each pixel was determined only by the information of the pixel itself and its neighborhood.
– Thus, those methods are called ”local” or “area-based” methods.
• Global methods find better solutions in expense of more computations
– Optimize jointly the disparity values of all the pixels of each scanline (e.g. Dynamic Programming) – Optimize jointly the disparity values of all the pixels of the image (e.g. graph cuts)
DTU Electrical Engineering
Global Stereo Correspondence
• Up to this point, the disparity of each pixel was determined only by the information of the pixel itself and its neighborhood.
– Thus, those methods are called ”local” or “area-based” methods.
• Global methods find better solutions in expense of more computations
– Optimize jointly the disparity values of all the pixels of each scanline (e.g. Dynamic Programming) – Optimize jointly the disparity values of all the pixels of the image (e.g. graph cuts)
• In global algorithms, stereo correspondence is formulated as an energy function minimization problem, consisting of data and smoothness terms.
DTU Electrical Engineering
• What is Stereo Vision?
• Stereo/Epipolar Geometry
• Rectified Stereo Case
• Depth from Stereo Matches • Correspondence Problem
• Dense vs Sparse Correspondence • Local vs Global Correspondence
• (Dis)-Similarity Measures
DTU Electrical Engineering
• We discussed about what Stereo Vision is.
DTU Electrical Engineering
• We discussed about what Stereo Vision is.
• We learned about :
– Stereo/Epipolar Geometry
epipolar line
DTU Electrical Engineering
• We discussed about what Stereo Vision is.
• We learned about :
– Stereo/Epipolar Geometry – Rectified Stereo Case
DTU Electrical Engineering
• We discussed about what Stereo Vision is.
• We learned about :
– Stereo/Epipolar Geometry – Rectified Stereo Case
– Depth from Stereo Matches
Reference Image
Disparity Map
DTU Electrical Engineering
• We discussed about what Stereo Vision is.
• We learned about :
– Stereo/Epipolar Geometry – Rectified Stereo Case
– Depth from Stereo Matches
• Correspondence Problem
• Dense vs Sparse Correspondence • Local vs Global Correspondence
• (Dis)-Similarity Measures
Stereo Vision
Sparse output
Dense output
•Local Methods (Area-based) •Global Methods (Energy-based) •Other Methods
DTU Electrical Engineering
Stereo Vision
DTU Electrical Engineering
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com