Fundamentals of Computer Vision
Mohamed Almekkawy
School of Electrical Engineering and Computer Science Penn State University – CMPEN/EE 454
Today
• Course overview
• Course logistics
• What is computer vision?
Course Goals and Objectives
• Introduce the fundamental problems of computer vision.
• Introduce the main concepts and techniques used to solve those problems.
• Enable students to implement vision algorithms
• Enable students to make sense of the vision literature
Optional Textbooks
Computer Vision: Algorithms and Applications, Rick Szeliski, Springer, 2011. http://szeliski.org/Book/
Computer Vision: Models, Learning,
and Inference, Simon Prince,
Cambridge Univ Press, 2012. http://www.computervisionmodels.com
Optional Textbooks
Concise Computer Vision
https://www.springer.com/gp/boo k/9781447163190
Introduction to Deep Learning
https://www.springer.com/gp/boo k/9783319730035
Optional Textbooks
Introductory Techniques for 3-D Computer Vision Computer Vision: A Modern Approach, by by E. Trucco and A. Verri, Prentice Hall, 1998. D.Forsyth and J.Ponce, Prentice Hall, 2002.
Course Requirements
• Prerequisites
• A good working knowledge of programming, especially programming in MATLAB
• Linear algebra • Vector calculus
• Course does NOT assume prior imaging experience, computer vision, image processing, graphics, etc.
Grading:
• Homework: • Projects:
• Exam 1:
• Exam 2:
• Exam 3:
20% 30% 20% 20% 10 %
Spring 2020 Syllabus (excerpts)
Individual exams and assignments are not scaled/curved, but we do scale the accumulated numeric course scores before assigning a letter grade at the end of the course.
See full syllabus posted on Canvas course web site
What is computer vision?
What a person sees
What a computer sees
The goal of computer vision is to give computers
(super) human-level perception
Typical Perception Pipeline
Representation
‘fancy math’
Output
Typical Perception Pipeline
Representation
‘fancy math’
output
what should we look at? (image features)
what can we understand? (semantic segmentation)
Typical Perception Pipeline
representation
‘fancy math’
Output
what should we look at? (image features)
what can we understand? (semantic segmentation)
Typical Perception Pipeline
Representation
‘fancy math’
Output
what should we look at? (image features)
what can we understand? (semantic segmentation)
Every image tells a story
• Goal of computer vision: perceive the “story” behind the picture
• Compute properties of the world
• • •
3D shape
Names of people or objects
What happened?
The goal of computer vision
Can the computer match human perception?
• Yes and no (mainly no)
– computers can be better at “easy” things – humans are much better at “hard” things
• But huge progress has been made
– Especially in the last 10 years
– What is considered “hard” keeps changing
Why study Computer Vision?
• Images and movies are everywhere
• Fast-growing collection of useful applications
• building representations of the 3D world from pictures
• automated surveillance (who’s doing what)
• movie post-processing
• face finding
• Various deep and attractive scientific mysteries
• how does object recognition work?
• Greater understanding of human vision
Computer Vision – A Modern Approach Set: Introduction to Vision Slides by D.A. Forsyth
Industry
Paid sponsors for CVPR’16
Applications of computer vision
Automated visual inspection
Machine Vision
Medical imaging
3D imaging MRI, CT
Image guided surgery Grimson et al., MIT
Source: S. Seitz
Industry Applications
Tesla Google BMW Mercedes Uber
Autonomous driving
…
Night vision
“Around view” camera
Vision in Cars
Object Recognition
Toshiba Tech IS-910T
DataLogic LaneHawk LH4000
2013
2012
Face Detection
Image Stitching
Photosynth (Analyzes Digital Photograph)
Tango
Computer Vision for VR
It is a good time to do computer vision
Current state of the art
• You just saw examples of current systems. – Many of these are less than 5 years old
• This is a very active research area, and rapidly changing – Many new apps in the next 5 years
• To learn more about vision applications and companies
– David Lowe maintains an excellent overview of vision companies • http://www.cs.ubc.ca/spider/lowe/vision.html
Why is Computer Vision Hard?
If we already know the geometry, surface material and lighting conditions, it is well-understood how to generate the value at each pixel. [this is Computer Graphics]
But this confluence of factors contributing to each pixel can not be easily decomposed. The process can not be inverted.
Tell me what do you see?
Vision is an ill-posed mathematical problem.
Ames Room Demystified
Shape of room doesn’t conform to our assumption of a rectangular parallelepiped (box shape)
Ames Room Demystified
Tentative Topics to be covered
Image processing:
• Basics of filtering.
• Image pyramids.
• Gradients and lines.
• Hough transforms.
Tentative Topics to be covered
Feature detection and correspondences:
• Corner detection.
• SIFT et al.
• Feature descriptors.
• RANSAC.
Tentative Topics to be covered
Transformations and geometry:
• Homographies and image alignment.
• Camera models.
• Fundamental matrix.
• Epipolar geometry and stereo.
• Structure from motion.
Tentative Topics to be covered
Physics-based vision:
• Reflectance and image formation.
• Shape from shading.
• Color.
Tentative Topics to be covered
Objects, faces, and learning:
• Basics of probability.
• CNN, PCA.
• Face detection.
• Perceptron, backpropagation.
Tentative Topics to be covered
Dealing with motion:
• Optical flow.
• Image registration.
• Kalman Filtering.
• Tracking.