Mosaics (homographies and blending)
© Jeffrey Martin (jeffrey-martin.com)
with a lot of slides stolen from Steve Seitz and Rick Szeliski
Slides from Alexei Efros, CMU
Why Mosaic?
Are you getting the whole picture? • Compact Camera FOV = 50 x 35°
Slide from Brown & Lowe
Why Mosaic?
Are you getting the whole picture?
• Compact Camera FOV = 50 x 35°
• Human FOV = 200 x 135°
Slide from Brown & Lowe
Why Mosaic?
Are you getting the whole picture?
• Compact Camera FOV = 50 x 35°
• Human FOV = 200 x 135°
• Panoramic Mosaic = 360 x 180°
Slide from Brown & Lowe
Mosaics: stitching images together
virtual wide-angle camera
Basic camera model
image plane
“equivalent” image plane
lens
Basic camera model
“image plane”
optical center
camera direction
“image plane” if camera is turned
For simplicity, we will draw image planes in front of the optical center
A pencil of rays contains all views
real camera
synthetic camera
Can generate any synthetic camera view
as long as it has the same center of projection!
How to do it?
Basic Procedure
• Take a sequence of images from the same position – Rotate the camera about its optical center
• Compute transformation between second image and first
• Transform the second image to overlap with the first
• Blend the two together to create a mosaic
• If there are more images, repeat
…but wait, why should this work at all?
• What about the 3D geometry of the scene?
• Why aren’t we using it?
Aligning images
left on top
right on top
Translations are not enough to align the images
Image reprojection
mosaic PP
The mosaic has a natural interpretation in 3D
• The images are reprojected onto a common plane
• The mosaic is formed on this plane
• Mosaic is a synthetic wide-angle camera
Image reprojection
Basic question
• How to relate two images from the same camera center? – how to map a pixel from PP1 to PP2
Answer
• Cast a ray through each pixel in PP1
• Draw the pixel where that ray intersects PP2
But don’t we need to know the geometry of the two planes in respect to the eye?
PP1
PP2
Observation:
Rather than thinking of this as a 3D reprojection,
think of it as a 2D image warp from one image to another
Back to Image Warping
Which t-form is the right one for warping PP1 into PP2? e.g. translation, Euclidean, affine, projective
Translation
Affine Perspective
6 unknowns 8 unknowns
2 unknowns
Central Projection and Homographies
Central projection: mapping between any two PPs with the same center of projection based on ray-correspondences
• preserves straight lines (Why?)
• parallel lines aren’t
• rectangle should map to arbitrary quadrilateral
Since straight lines are preserved, it can be
described by a homographic transformation (remember general property of homographies from topic 1)
PP2
wx’ * * * x
PP1
wy’ * * *y w ***1
p’ H p
using homogeneous representation of 2D points w.r.t. any given coordinate systems in each plane
Image warping with homographies
image plane in front
image plane below
black area where no pixel maps to
Image rectification
p
p’
To unwarp (rectify) an image
• Find the homography H given a set of p and p’ pairs
• How many correspondences are needed?
• Tricky to write H analytically, but we can solve for it!
• Find such H that “best” transforms points p into p’
• Use least-squares if more than 4 point correspondences
Computing Homography Consider one point correspondence
p(x,y) p’(x’, y’)
wx’ a b cx wy’ d e f y
p’Hp
x’ gx hy i gx hy i in Hartley and
Two equations linear w.r.t unknown coefficients of matrix H and quadratic w.r.t. known point coordinates (x,y,x’,y’)
w g h i1
axbyc axbycgxx’hyx’ix’0
See p.35
y’ dxey f
Zisserman
dxey f gxy’hyy’iy’0
Computing Homography Consider 4 point correspondences
pi(xi,yi) p’i(x’i,y’i) wx’ a b cx
p’Hp i i i ii wy’d e fy
fori=1,2,3,4
Special case of DLTmethod
iii w g h i1
i ax by cgxx’ hyx’ ix’ 0
Can be written in matrix multiplication form Ai h 0 for i=1,2,3,4
ii iiiii (seep.89
dxi eyi f gxiy’ihyiy’iiy’i 0
in Hartley and Zisserman)
where h [ a b c d e f g h i ]T is a vector of unknown coefficients in H and Ai is a 2×9 matrix based on known point coordinates xi , yi , x’i , y’i
Computing Homography Consider 4 point correspondences
pi(xi,yi) p’i(x’i,y’i)
p’i Hpi Ai h0 2×9 9×1 2×1
A1 A2
fori=1,2,3,4
All four matrix equations can be put into one equation
A h 0 3 8×1
or
A4 8×9
Ah0
8×9 9×1 8×1
Computing Homography Consider 4 point correspondences
p’i Hpi for i=1,2,3,4
pi(xi,yi) p’i(x’i,y’i) (*)
Questions: 8 linear equations, 9 unknowns?
One possible approach to solving problem (*):
set value for one of coefficients in h to 1, e.g.
i=1
(Why OK?)
Does not work
if i=0 for the
solution of (*) (this happens iff coordinate center p = (0,0) maps
Ah0
8×9
9×1 8×1
trivial solution h=0?
A1:8 h1:8 A9
8×8
8×1 8×1
first 8 columns of A
first 8 rows of h
9th columns of A
to point p’ at infinity)
Computing Homography Consider 4 point correspondences
pi(xi,yi) p’i(x’i,y’i) (*)
p’i Hpi for i=1,2,3,4
Questions:
What if 4 points correspondences are known with error?
Are there any benefits from knowing more point correspondences?
Will look at this in the next topic…
Ah0
8×9 9×1
8×1
Fun with homographies Original image
Virtual camera rotations
Panoramas
1. Pick one image (red)
2. Warp the other images towards it (usually, one by one)
3. blend
changing camera center Does it still work?
PP1
synthetic PP
What color?
is not even visible in PP1
PP2
Planar scene (or far away) PP3
PP1
PP3 is a projection plane of both centers of projection, so we are OK!
This is how big aerial photographs are made
PP2
Planar mosaic
Blending the mosaic
An example of image compositing: the art (and sometime science) of combining images together…
Feathering
+
11 00
=
Encoding transparency I(x,y) = (aR, aG, aB, a)
Iblend = Ileft + Iright
Setting alpha: simple averaging
Alpha = .5 in overlap region
Image feathering
Weight each image proportional to its distance from the edge
(distance map [Danielsson, CVGIP 1980]
1. Generate weight map for each image
2. Sum up all of the weights and divide by sum: weights sum up to 1: wi’ = wi / ( ∑i wi)
Setting alpha: center seam
Distance transform
Alpha = logical(dtrans1>dtrans2)
Setting alpha: blurred seam
Distance transform
Alpha = blurred
Setting alpha: center weighting
Distance transform
Alpha = dtrans1 / (dtrans1+dtrans2)
Ghost!
Programming Project
Homographies and Panoramic Mosaics
• Capture photographs (and possibly video) • Might want to use tripod
• Compute homographies (define correspondences) • will need to figure out how to setup system of eqs.
• (un)warp an image (undo perspective distortion)
• Produce 3 panoramic mosaics (with blending)
• Do some of the Bells and Whistles
Bells and Whistles
Blending and Compositing
• use homographies to combine images or video and images together in an interesting (fun) way. E.g.
– put fake graffiti on buildings or chalk drawings on the ground – replace a road sign with your own poster
– project a movie onto a building wall
– etc.
3D Sidewalk Art by Julian Beever (see links for more)
Bells and Whistles
3D Sidewalk Art by Edgar Müller
Bells and Whistles
3D Sidewalk Art by Edgar Müller
Bells and Whistles
Capture creative/cool/bizzare panoramas • Example from UW (by Brett Allen):
Capture a 360 panorama
a bit trickier… projecting all images onto a common “reference” cylinder or sphere, rather than a plane
Bells and Whistles
Video Panorama
• Capture two (or more) stationary videos (either from the same point, or of a planar/far-away scene). Compute homography and produce a video mosaic. Need to worry about synchronization (not too hard).
• e.g. capturing a football game from the sides of the stadium
Other interesting ideas?
• talk to me
From previous year’s classes
Ben Hollis, 2004
Ben Hollis, 2004
Matt Pucevich , 2004
Eunjeong Ryu (E.J), 2004
Go Explore!
Ken Chu, 2004