Computer Vision (7CCSMCVI / 6CCS3COV)
Recap
• Image formation
● Low-level vision ● Mid-level vision
• •
grouping and segmentation (finding matching elements within an image)
correspondence problem (finding matching elements across images)
● for all locations or selected interest points
● comparing image intensities or descriptors
● finding matches by search
● determining similarity between elements
● dealing with false matches by model fitting
Stereo and Depth ←Today ● High-level vision
●
Computer Vision / Mid-Level Vision / Stereo and Depth
1
Today
• Stereo vision
– stereo camera geometry
» coplanar cameras (simple case)
» non-coplanar cameras (complex case)
– disparity measurement » calculating depth
– correspondence
» stereo constraints used to solve the correspondence problem
• Other cues to depth – Binocular
– Oculomotor – Monocular – Motion
Computer Vision / Mid-Level Vision / Stereo and Depth 2
Why is stereo vision important?
x1
2
z1
2
x ‘= f ‘ X 1= f ‘ X 2 Z1 Z2
A camera projects 3D points onto a 2D plane
3D points on the same line-of-sight have the same 2D image location (i.e. imaging results in depth information loss)
Computer Vision / Mid-Level Vision / Stereo and Depth
3
Why is stereo vision important?
Depth information can be recovered using two images and a knowledge of geometry.
e.g. all points P, P1, P2, and P3 project to the same location in the left image, but to different locations in the right image.
The right image allows us to measure how far each of these points are from the left camera (if we can solve the correspondence problem).
Computer Vision / Mid-Level Vision / Stereo and Depth 4
Why is stereo vision important?
This is useful for:
Path planning / collision avoidance (car / robot) virtual advertising
Not stereo, but same methods can be used for 3D model building
++
→
Computer Vision / Mid-Level Vision / Stereo and Depth
5
Stereo: coplanar cameras
OL
i
j
k
(x’L,y’L)
P = (x,y,z)
OR
i
j
Simplest Case
• Image planes of cameras are coplanar
• Focal lengths equal
• Optical centres are at same height (i.e. x-axes collinear)
• Intersection of optical axes at infinity (i.e. z-axes parallel)
k
(x’R,y’R)
Computer Vision / Mid-Level Vision / Stereo and Depth 6
baseline B
Image formation reminder Virtual image
P’
k
f
P = (x,y,z)
O
i
j
3D scene point P is projected to a point P’ on the image, such that:
P’=(x’,y’)=(fx, fy) zz
Assuming that the image centre is (0,0) [see lecture 2]
Computer Vision / Mid-Level Vision / Stereo and Depth
7
(x’,y’) (0,0)
Stereo: coplanar cameras
OL
i
j
k
(x’L,y’L)
P = (x,y,z) w.r.t. OL
OR
i
j
Image formation for two cameras:
P projects to (x’R, y’R) and (x’L, y’L)
Note: Because x-axes of cameras are collinear, y’L = y’R
k
(x’R,y’R)
Computer Vision / Mid-Level Vision / Stereo and Depth 8
baseline B
Stereo: coplanar cameras
OL
i
j
(x’R,y’R)
(x’L,y’L)
P = (x,y,z) w.r.t. OL
k
OR
i
j
k
(x’L,y’L)=(fx, fy) zz
(x’R,y’R)=(f(x−B), fy) zz
Using the coordinate system of the left camera (since xR=xL-B)
Computer Vision / Mid-Level Vision / Stereo and Depth 9
baseline B
Stereo: coplanar cameras
OL
i
j
(x’R,y’R)
(x’L,y’L)
P = (x,y,z) w.r.t. OL
k
OR
i
j
k
(x’L,y’L)=(fx, fy) zz
(x’R,y’R)=(f(x−B), fy) zz
Disparity, d=x’L−x’R= fx−f(x−B)= fB z=f Bd zzz
Computer Vision / Mid-Level Vision / Stereo and Depth
10
baseline B
Disparity
Depth is inversely proportional to disparity.
B d
then we can calculate the depth of a point.
Even without the baseline, we can know the relative depths of points from their relative disparities.
z=f
If the baseline distance is known, and we can measure the disparity,
Computer Vision / Mid-Level Vision / Stereo and Depth 11
Disparity
The difference vector of the image coordinates of two corresponding points.
NOTE
CORRESPONDING POINTS
DISPARITY VECTOR
● ● ● ●
the disparity, d, of a point is a 2D vector.
disparity is measured in pixels and can be positive or negative
a pair of stereo images defines a field of disparity vectors (a disparity map) For coplanar cameras disparity is horizontal only
Computer Vision / Mid-Level Vision / Stereo and Depth 12
SUPERIMPOSED IMAGES
Disparity / depth map: example
left image right image
depth map
light = close, dark = far
Computer Vision / Mid-Level Vision / Stereo and Depth
13
Disparity / depth map: example
left image
right image
False shallow region caused by false matches
Computer Vision / Mid-Level Vision / Stereo and Depth
14
depth map
light = close, dark = far
The stereo correspondence problem
To measure disparity, it is necessary to find corresponding points in the stereo pair of images.
To solve the stereo correspondence problem, we can use:
• Correlation-based methods
yield dense disparity maps: a disparity value at each pixel.
• Feature-based methods
yield sparse disparity maps: a disparity value at interest points
only.
Computer Vision / Mid-Level Vision / Stereo and Depth 15
The stereo correspondence problem
To measure disparity, it is necessary to find corresponding points in the stereo pair of images.
Basic requirements to be able to solve the correspondence problem: 1. Most scene points visible in both images
2. Corresponding image regions appear “similar”
These assumptions hold if:
• The distance of the 3D point from the cameras is much larger than the baseline: z >> B
Computer Vision / Mid-Level Vision / Stereo and Depth 16
The stereo correspondence problem
To measure disparity, it is necessary to find corresponding points in the stereo pair of images.
As we saw in the previous lecture, solving the correspondence problem is not easy.
However, we can use knowledge about the stereo camera system to help find a solution…
Computer Vision / Mid-Level Vision / Stereo and Depth 17
Stereo Constraints on Correspondence
Epipolar constraint
For coplanar cameras, y’L = y’R so 2D search can be reduced to a 1D search along the “epipolar” line (= the corresponding row of pixels for
coplanar cameras).
Computer Vision / Mid-Level Vision / Stereo and Depth 18
Stereo Constraints on Correspondence
Maximum disparity constraint
Length of search region depends on the maximum expected disparity, often predictable geometrically (dmax = fB/zmin).
x’ x’
For each point (x’, y’) in the left image, search for its corresponding point between (x’-dmax,y’) and (x’+dmax,y’) in the right image.
Computer Vision / Mid-Level Vision / Stereo and Depth 19
Stereo Constraints on Correspondence
Continuity
Neighbouring points should have similar disparities, because the environment is made of continuous surfaces over which depth varies smoothly.
Centre of left camera
object
Centre of right camera
Computer Vision / Mid-Level Vision / Stereo and Depth
20
right image left image
Stereo Constraints on Correspondence
Continuity
The exception is at discontinuities where depth (and hence disparity) can change suddenly.
object Centre of left camera
object
Centre of right camera
right image left image
Computer Vision / Mid-Level Vision / Stereo and Depth
21
Stereo Constraints on Correspondence
Uniqueness
A location in one image should only match a single location in the other image.
Centre of left camera
object
Centre of right camera
Computer Vision / Mid-Level Vision / Stereo and Depth
22
right image left image
Stereo Constraints on Correspondence
Uniqueness
The exception is when a surface lies along a line-of-sight for one camera (in this case one location may match many locations).
If fact any inclined surface may project to n pixels in one image and m pixels in the other image (with m ≠ n).
Centre of left camera
object
Centre of right camera
right image left image
Computer Vision / Mid-Level Vision / Stereo and Depth
23
Stereo Constraints on Correspondence
Ordering
Matching points along corresponding epipolar lines should be in the same order.
Centre of left camera
object
Centre of right camera
Computer Vision / Mid-Level Vision / Stereo and Depth
24
right image left image
Stereo Constraints on Correspondence
Ordering
The exception is when objects have different depths.
Centre of left camera
object
Centre of right camera
right image left image
Computer Vision / Mid-Level Vision / Stereo and Depth
25
Correspondence problems
Correspondence is fundamentally ambiguous, i.e. there are many possible solutions.
Centre of left camera
object
Centre of right camera
Computer Vision / Mid-Level Vision / Stereo and Depth
26
right image left image
Correspondence problems
Correspondence is fundamentally ambiguous, i.e. there are many possible solutions.
Centre of left camera
object
Centre of right camera
right image left image
Computer Vision / Mid-Level Vision / Stereo and Depth
27
Correspondence problems
Correspondence is fundamentally ambiguous, i.e. there are many possible solutions.
Centre of left camera
object
Centre of right camera
Computer Vision / Mid-Level Vision / Stereo and Depth
28
right image left image
Correspondence problems
Correspondence is fundamentally ambiguous, i.e. there are many possible solutions.
We are trying to use imperfect constraints to narrow down these many potential solutions to the correct one.
Centre of left camera
object
Centre of right camera
Computer Vision / Mid-Level Vision / Stereo and Depth
29
right image left image
Correspondence problems
Some points in each image will have no corresponding points in the other image:
1. due to occlusion (e.g. )
2. the cameras might have different fields of view (e.g. )
Centre of left camera
object
Centre of right camera
right image left image
Computer Vision / Mid-Level Vision / Stereo and Depth
30