Image Formation and the Image Processing Pipeline
18-793 Summer 2020
Slide-deck Credit
Srinivasan Narasimham (CMU) Fredo Durand (MIT)
Mark Levoy (Stanford)
Ashok Veeraraghavan (Rice)
this photo
how did this scene
become
Digital processing
Two Pieces
Image formed on the sensor
scene
1. How is the image formed on the sensor ?
2. How is the sensed image readout and processed ?
camera
Image Formation
Pinhole Cameras
Image credit: Arielle Live
Image credit:http://facweb.cs.depaul.edu/sgrais/pinhole_cameras.htm
Image credit: Ina Marie Schmidt
Image credit: Alemonio
• Fullysharpimage!
• Dimming near the edge
Pinhole optics
s – size of the sensor
f
– pinhole to sensor distance, focal length
s
fd
Pinhole optics
Film/
sensor pinhole scene
• Varyingfocallength changes field-of-view
Film/ sensor
pinhole
scene
s
fd 2f
Pinhole optics
• Varyingfocallength changes field-of-view
Film/ sensor
pinhole
scene
s
fd 2f
Pinhole optics
Frédo Durand — MIT Computer Science and Artificial Intelligence Laboratory –
fredo@mit.edu
Vignetting
Pinhole
C
B A
Image credit: Alemonio
• Whathappensifwe increase the size of the pinhole ?
– Geometric blur
Pinhole optics
Image credit: David Heeger
Slide credit: Levoy, London (image)
Slide credit: Levoy, London (image)
Pinhole optics
• Soisdecreasingpinholealwaysgood?
Pinhole optics
• Soisdecreasingpinholealwaysgood?
Image credit: David Heeger
Credit: Levoy
• Largepinhole (geometric blur)
• Smallpinhole (diffraction)
• Optimalpinhole (not enough light)
Credit: Hecht, Levoy
Pinhole optics
Wavelength of light
Credit: wikipedia
Credit: Levoy
Credit: Levoy
Lens-based imaging
• Focus/focusplane
• Defocusblur
• Depthoffield(DoF)
• RelationshipbetweenDoFandaperturesize • Relationshipbetweenapertureandexposure
Focus plane
Thin lens law
Sensor
Changing Focus
Point in defocus
Point in focus
Focus plane
Defocus
Sensor
Focus plane
Defocus
Sensor
Focus plane
Defocus
Sensor
Focus plane
Defocus
Sensor
Changing Focus
Point in defocus
Point in focus
Focus plane
Magnification
Field of view (when focused at infinity) = tan
Magnification:
Sensor
Larger focal length = larger magnification (telephoto lens) = (smaller FoV) Smaller focal length = larger FoV (wideangle lens) = (smaller magnification)
Focal Length vs. viewpoint
• FocallensdoesNOTONLYchangesubjectsize • Samesizebymovingtheviewpoint
• Differentperspective(e.g.background)
Slide Credit: Fredo Durand, CSAIL, MIT
Focal Length vs. viewpoint
Telephoto makes it easier to
select background (a small change in viewpoint is a big change in background)..
Slide Credit: Fredo Durand, CSAIL, MIT
Changing Aperture
Depth of field
Focus plane
Depth of field
Sensor
Focus plane
Depth of field
Sensor
pixel width
Focus plane
Depth of field
Sensor
pixel width
Focus plane
Depth of field
All points within this volume will appear in sharp focus
Depth of field
Sensor
pixel width
Depth of field
Depth of field
N = f/D is referred to as the F/#
Depth of field
Focus plane
focal length f diameter d
Depth of field
𝐷𝑂𝐹 2𝑊𝑁𝑣 𝑓
All points within this volume will appear in sharp focus
N = f/d is referred to as the F/# describes the size of the aperture
Depth of field
Sensor
pixel width W
Focus plane
focal length f diameter d
Depth of field
Sensor
pixel width W
Depth of field and aperture
Frédo Durand — MIT CSAIL fredo@mit.edu
Depth of field and focus distance
Frédo Durand — MIT CSAIL fredo@mit.edu
Depth of field and focal length
Frédo Durand — MIT CSAIL fredo@mit.edu
Image credit
Augustine’s Notebook
http://audster.wordpress.com/tag/cannon/
Depth of field
Image Credit: elementsofcinema.com
Depth of field
Image credit: RHiNO NEAL
Depth of field
Image credit:
Image credit: Mr.Kitux
Aperture (f/#)
Recall, larger aperture = smaller f/# (N = f/D) Depth of field is linearly proportional to f/# Doubling f/# doubles the depth of field.
Slide Credit: Marc Levoy
• Exposure is proportional to the time the shutter is open
• Expressed in fraction of a second (1/60s, 1/125s, 1/250s, 1/500s, etc.)
Shutter
• Most of the time, the film/sensor is protected from light
• When we take a picture, the shutter opens and closes, thereby exposing the film.
Frédo Durand — MIT CSAIL fredo@mit.edu
Effect of shutter speed
• Longer shutter speed => more light, but more motion blur
• Faster shutter speed freezes motion
Frédo Durand — MIT CSAIL fredo@mit.edu
Effect of shutter speed
• Longer shutter speed => more light, but more motion blur
• Faster shutter speed freezes motion
Frédo Durand — MIT CSAIL fredo@mit.edu
Shutter speed and focal length • Because telephoto “magnify”, they also magnify your
hand shaking
• Telephotos therefore require faster shutter speed
• Solution: Image stabilization
– mechanically compensates for vibration
Frédo Durand — MIT CSAIL fredo@mit.edu
Your best friend
• Use a tripod! It will always enhance sharpness
Frédo Durand — MIT CSAIL fredo@mit.edu
Frédo Durand — MIT CSAIL fredo@mit.edu
Exposure and Aperture
• Two main parameters: – Aperture (in f stop)
– Shutter speed (in fraction of a second)
Frédo Durand — MIT CSAIL fredo@mit.edu
f/2.0 to f/4.0?
Aperture
• Diameter of the lens opening (controlled by diaphragm)
• Expressed as a fraction of focal length, in f-number
– f/2.0 on a 50mm means that the aperture is 25mm
– f/2.0 on a 100mm means that the aperture is 50mm
• Disconcerting: small f number = big aperture
• What happens to the area of the aperture when going from
• Typical f numbers are
f/2.0, f/2.8, f/4, f/5.6, f/8, f/11, f/16, f/22, f/32
– See the pattern?
Frédo Durand — MIT Computer Science and Artificial Intelligence Laboratory – fredo@mit.edu
Slide credit: Marc Levoy
Common lens related issues
Slide-deck credit: Srinivasan Narasimhan (CMU)
Vignetting
L3 L2 L1
B
More light passes through lens L3 for scene point A than scene point B
Results in spatially non-uniform brightness (in the periphery of the image)
A
Vignetting
Usually brighter at the center and darker at the periphery.
Reading: http://www.dpreview.com
Vignetting
photo by Robert Johnes
Chromatic Aberration
longitudinal chromatic aberration transverse chromatic aberration (axial) (lateral)
Chromatic Aberrations
longitudinal chromatic aberration transverse chromatic aberration (axial) (lateral)
Chromatic Aberrations
Reading: http://www.dpreview.com
Lens Glare
Stray interreflections of light within the optical lens system. Happens when very bright sources are present in the scene.
Reading: http://www.dpreview.com
Geometric Lens Distortions
Radial distortion Tangential distortion
Both due to lens imperfection Rectify with geometric camera calibration
Photo by Helmut Dersch
Radial Lens Distortions
No Distortion Barrel Distortion Pincushion Distortion
• Radial distance from Image Center: ru =rd +k1rd3
Correcting Radial Lens Distortions
Before After
http://www.grasshopperonline.com/barrel_distortion_correction_software.html
Image credit: Katrina (Spiritus Capere)
Lens bokeh
Image credit: Ken Rockwell
Image credit: Ken Rockwell
Image credit: Ken Rockwell
Image credit: Ken Rockwell
Image credit: Wikimedia
Image Processing
Dark Current Noise Subtraction
Original image + Dark Current Noise Image with lens cap on
Result of subtraction
Copyright Timo Autiokari, 1998-2006
Post-processing
Slide Credits
Most slides in this slide deck are from similar courses at Stanford and MIT by Prof. Marc Levoy and Prof. Fredo Durand.