Computer Vision
Camera: Perspective projection
Semester 1
Changjae Oh
Copyright By PowCoder代写 加微信 powcoder
Announcement of
Pope Benedict XVI (2005),
Pope Francis (2013)
• The pinhole projection model ̶ Geometric properties
̶ Perspective projection matrix
• Cameras with lenses
Depth of focus Field of view Lens aberrations
• Digital sensors
• The pinhole projection model ̶ Geometric properties
̶ Perspective projection matrix
• Cameras with lenses
Depth of focus Field of view Lens aberrations
• Digital sensors
Let’s design a camera
• Idea 1: put a piece of film in front of an object • Do we get a reasonable image?
Pinhole camera
• Add a barrier to block off most of the rays
̶ Tiny aperture without lens
̶ Light from a scene passes through the aperture and projects an inv erted image on the film
Pinhole camera
Captures pencil of rays
All rays through a single point: aperture, center of projection, optic al center, focal point, camera center
The image is formed on the image plane (film)
Pinhole camera
f = focal length
c = center of the camera
Slide from Forsyth
Pinhole cameras are everywhere
Tree shadow during a solar eclipse
photo credit: Nils van der Burg
http://www.physicstogo.org/index.cfm
• Basic principle known to Mozi (470-390 BCE), Aristotle (384-322 BCE)
• Drawing aid for artists: described by Le onardo da Vinci (1452-1519)
Image source
Slide from Lazebnik
Turning a room into a camera
• A. Torralba and W. Freeman, Accidental Pinhole and Pinspeck Cameras, CVPR 2012
Turning a room into a camera obscura
Accidental Pinhole and Pinspeck Cameras Revealing the scene outside the picture. , . Freeman
Turning a room into a camera obscura
Pinhole projection model
• To compute the projection P’ of a scene point P,
̶ Form the visual ray connecting P to the camera center O and find where it intersects the image plane
Pinhole projection model
The coordinate system
The optical center (O) is at the origin
The image plane is parallel to xy-plane or perpendicular to the z-axis, which is the optical axis
Pinhole projection model
• Projection equations
• Derived using similar triangles
( x , y , z ) → ( f xz , f yz )
Dimensionality reduction: from 3D to 2D
3D world 2D image
Point of observation
What properties of the world are preserved? • Straightlines,incidence
What properties are not preserved? • Angles,lengths
Slide by A. Efros
Properties of projection
Who is taller?
Which is closer?
Slide by Derek Hoiem
Properties of projection
Perpendicular?
Slide by Derek Hoiem
What happens to the projection of a pattern on a plane parallel to the image plane?
All points on that plane are at a fixed depth z
The pattern gets scaled by a factor of f / z, but angles and ratios of lengths/areas are
parallel planes
Point of observation
( x , y , z ) → ( f xz , f yz )
parallel planes
What happens to the projection of a pattern on a plane parallel to the image plane?
All points on that plane are at a fixed depth z
The pattern gets scaled by a factor of f / z, but angles and ratios of lengths/areas are
Piero della Francesca, Flagellation of Christ, 1455-1460 , The Music Lesson, 1662-1665
Slide from S. Lazebnik
Vanishing points
• All parallel lines converge to a vanishing point
̶ Each direction in space is associated with its own vanishing point ̶ Exception: directions parallel to the image plane
Constructing the vanishing point of a line
image plane
vanishing point
camera center
line in the scene
Slide from S. Lazebnik
Vanishing lines of planes
How do we construct the vanishing line of a plane?
Image source
Vanishing lines of planes
• Horizon: vanishing line of the ground plane
– All points at the same height as the camera project to the horizon
– Points higher (resp. lower) than the camera project above (resp. b elow) the horizon
– Provides way of comparing height of objects
camera center
plane in the scene
Slide by S. Seitz
Vanishing lines of planes
Is the parachutist above or below the camera?
Slide by S. Seitz
Comparing heights
Vanishing Point
Slide by S. Seitz
Measuring height
Camera height
What is the height of the camera?
Slide by S. Seitz
Perspective geometry
Illusion Credit: RN Shepard, Mind Sights: Original Visual Illusions, Ambiguities, and other Anomalies
Perspective cues in art
• Masaccio, Trinity, Santa Maria N ovella, Florence, 1425-28
• One of the first consistent uses o f perspective in Western art
Slide by S. Seitz
Perspective distortion
• What is the shape of the projection of a sphere?
Image source: F. Durand
Perspective distortion
• What is the shape of the projection of a sphere?
Slide from S. Lazebnik
Perspective distortion
• Are the widths of the projected columns equal?
The exterior columns are wider
This is not an optical illusion, and is not due to lens flaws Phenomenon pointed out by
Source: F. Durand
Perspective distortion: People
Slide from S. Lazebnik
Modelling projection:
world to image
• Projection equation:
Note: instead of dealing with an image that is upside down, most of the time we will pretend that the image plane is in front of the camera center.
( x , y , z ) → ( f xz , f yz )
Source: J. Ponce, S. Seitz
Homogeneous coordinate
• Nonlinearity in the projection equation
(x, y, z) → ( f xz , f yz ) : division by z is nonlinear
• Add one more coordinate for linear transformation
homogeneous image homogeneous scene coordinates coordinates
Converting from homogeneous coordinates
Homogeneous coordinate • Why does this matter?
Homogeneous coordinates can handle general cases Invariant to scaling
Point in Cartesian is ray in Homogeneous
x kx ky=ky kw = w
ky y
w kw kw w
Homogeneous Cartesian Coordinates Coordinates
Credit: J. Hays
Homogeneous coordinate
• EBU6230 Image and Video Processing
̶ Week 2 day 1: image_transformations
• https://pages.mtu.edu/~shene/COURSES/cs3621/NOTES/geometry/ho mo-coor.html
Perspective projection matrix • Projection:
a matrix multiplication with homogeneous coordinates
f 0 0 0x fx
0 f 0 0y=fy (fx,fy)
0 0 1 0z z z z
1 divide by the third co
ordinate In practice: lots of coordinate transformations…
2D point (3×1)
Camera to pixel coord. trans. matrix
Perspective projection matrix (3×4)
World to camera coord. trans. matrix
3D point (4×1)
Slide from S. Lazebnik
Orthographic Projection
• Special case of perspective projection
̶ Distance from center of projection to image plane is infinite ̶ Also called “parallel projection”
Orthographic Projection
• Special case of perspective projection
̶ Distance from center of projection to image plane is infinite ̶ Also called “parallel projection”
Slide from S. Lazebnik
Orthographic Projection
• Special case of perspective projection
Distance from center of projection to image plane is infinite Also called “parallel projection”
Image World
What’s the projection matrix?
• When the 3D world dimension reduces to 2D, what properties are preserved? ̶ Straight lines
̶ Incidence
• In the figure below, there are two blue parallel lines, perpendicular to the image plane. Draw how the lines look like in the image plan and discuss the result
Constructing the vanishing point of a line
image plane
camera center
lines in the scene
• Perform Orthograpic projection to each 3D cube in 3D below and show the result in 2D
03) Orthographic projection
(0,1,0) (0,0,0) (-1,1,0)
04) Projection matrix (2D
̶ 𝒙= 𝒙,…,𝒙 = 𝑢,𝑣 ,…,(𝑢,𝑣) = 0,260, 640,260, 0,400, 640,400 141144
̶ 𝒙′ = 𝒙′1,…,𝒙′4 = 𝑢′1,𝑣′1 ,…,(𝑢′4,𝑣′4) = 0,0 , 400,0 , 0,640 , 400,640 Find the perspective projection matrix 𝑷 such that 𝒙′ = 𝑷𝒙
Given two point sets:
Computer Vision
Camera: lenses & digital sensors
Semester 1
Changjae Oh
• The pinhole projection model ̶ Geometric properties
̶ Perspective projection matrix
• Cameras with lenses
Depth of focus Field of view Lens aberrations
• Digital sensors
Building a Real Camera
made pinhole camera
http://www.debevec.org/Pinhole/
Slide by A. Efros 52
Shrinking the aperture
• Images with varying the aperture size
• Why not make the aperture as small as possible? ̶ Less light gets through
̶ Diffraction effects…
Shrinking the aperture
• Images with varying the aperture size
Adding a lens
• A lens focuses light onto the film
̶ Thin lens model:
• Rays passing through the center are not deviated (pinhole projection model still holds)
Adding a lens
• A lens focuses light onto the film
̶ Thin lens model:
• Rays passing through the center are not deviated (pinhole projection model still holds)
• All rays parallel to the optical axis pass through the focal point
• All parallel rays converge to points on the focal plane
focal point
Thin lens formula
• Where does the lens focus the rays coming from a given point in the scene?
image plane
lens object
Slide by F. Durand
Thin lens formula
• What is the relation between the focal length ( f ),
the distance of the object from the optical center ( D ), and the distance at which the object will be in focus ( D′ )?
image plane
lens object
Slide by F. Durand
Thin lens formula
• Similar triangles everywhere!
y′/y = D′/D
image plane
lens object
Slide by F. Durand
Thin lens formula
• Similar triangles everywhere!
y′/y = D′/D y′/y = (D′−f )/f
image plane
lens object
Slide by F. Durand
Thin lens formula
• Any point satisfying the thin lens equation is in focus.
• What happens when D is very large?
y′/y = D′/D y′/y = (D′−f )/f
1+1=1 D′ D f
image plane
lens object
Slide by F. ′ D f
Depth of Field
• For a fixed focal length, there is a specific distance at which objects are “in focus”
̶ Other points project to a “circle of confusion” in the image
“circle of confusion”
Depth of Field
http://www.cambridgeincolour.com/tutorials/depth-of-field.htm
Slide by A. Efros
Controlling depth of field
• Changing the aperture size affects depth of field
̶ A smaller aperture increases the range in which the object is approximately in focus ̶ But small aperture reduces amount of light – need to increase exposure
http://en.wikipedia.org/wiki/File:Depth_of_field_illustration.svg
Varying the aperture
Large aperture = small DOF Small aperture = large DOF
Slide by A. Efros
Field of View
Slide by A. Efros
Field of View
Slide by A. Efros
Field of View
FOV depends on focal length and size of the camera retina
Larger focal length = smaller FOV
Slide by A. Efros
Field of View / Focal Length
Large FOV, small f Camera close to car
Small FOV, large f
Camera far from the car Sources: A. Efros, F. Durand
Same effect for faces
wide-angle standard telephoto
Source: F. Durand
Approximating an orthographic camera
Source: Hartley & Zisserman
The dolly zoom
• Continuously adjusting the focal length while the camera moves away fr om (or towards) the subject
http://en.wikipedia.org/wiki/Dolly_zoom
The dolly zoom
• Continuously adjusting the focal length while the camera moves away fr om (or towards) the subject
Example of dolly zoom from Goodfellas (YouTube)
Real lenses
Lens flaws: Vignetting
Radial Distortion
• Caused by imperfect lenses
• Deviations are most noticeable near the edge of the lens
No distortion Pin cushion Barrel
Lens flaws: Spherical aberration
• Spherical lenses don’t focus light perfectly
• Rays farther from the optical axis focus closer
Lens flaws: Chromatic Aberration
• Lens has different refractive indices for different wavelengths: causes co lor fringing
Near Lens Center Near Lens Outer Edge
• The pinhole projection model ̶ Geometric properties
̶ Perspective projection matrix
• Cameras with lenses
Depth of focus Field of view Lens aberrations
• Digital sensors
First digitally scanned photograph • 1957, 176×176 pixels
http://listverse.com/history/top-10-incredible-early-firsts-in-photography/
Camera sales over time
Camera sales over time • The full chart…
Digital cameras
• Digital cameras are not designed to be light measuring devices
• They are designed to produce visually pleasing photographs
• There is a great deal of processing (photo finishing) applied in the camera h ardware
Modern photography pipeline
Source: M. Brown
Digital camera sensors
• Each cell in a sensor array is a light-sensitive diode that converts photon s to electrons
̶ Dominant in the past: Charge Coupled Device (CCD)
̶ Dominant now: Complementary Metal Oxide Semiconductor (CMOS)
http://electronics360.globalspec.com/article/9464/ccd-vs-cmos-the-shift-in-image-sensor-technology
What does a raw image look like?
Color filter arrays
Bayer grid
Demosaicing:
Estimation of missing components from nei ghboring values
Why more green?
Human Luminance Sensitivity Function
Virtual Background
Demosaicing
• Producing full RGB image from mosaiced sensor output
Interpolate from neighbors:
Bilinear interpolation (needs 4 neighbors).
Bicubic interpolation (needs more neighbors, may overblur).
• Edge-awareinterpolation. • Large area of research.
Slide credit: I. Gkioulekas
Demosaicing
• Bilinear interpolation: Simply average your 4 neighbors.
• Neighborhood changes for different channels:
Slide credit: I. Gkioulekas
Digital camera artifacts • Noise
low light is where you most notice noise light sensitivity (ISO) / noise tradeoff stuck pixels
• In-camera processing
̶ oversharpening can produce halos
• Compression
̶ JPEG artifacts, blocking
• Blooming
̶ CCD charge overflowing into neighboring pixels
• Color artifacts ̶ Color moire
̶ Purple fringing from microlenses
Walking through the pipeline
Walking through the pipeline
Walking through the pipeline
Walking through the pipeline
• A Bayer filter mosaic is a color filter array (CFA) for arranging RGB color filters on a square grid of photose nsors. Figure 1 shows a raw image obtained by the Bayer pattern CFA where each pixel includes the R or G or B value. Perform image demosaicing to the image region in orange box using bilinear interpolation and provide each pixel’s RGB value, (R, G, B) of the 3 × 3 image.
Next topic
• How can we remove artifacts in images? ̶ Prerequisite
• Review EBU6230 Image/Video Processing – Week2: Image Filtering
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com