CS代考 Perception for Autonomous Systems 31392:

Perception for Autonomous Systems 31392:
Camera Matrix and Camera Calibration
Lecturer: —PhD
10 Feb. 2020 DTU Electrical Engineering 2

• Image formation
• Camera model – Project form World to sensor • Camera calibration – The 3D case
• Camera calibration – The realistic case
• Camera radial distortion
10 Feb. 2020 DTU Electrical Engineering 3

Image Formation
• Let’s look at
the most simplistic case. What’s weird about it?
10 Feb. 2020 DTU Electrical Engineering 4

Image Formation
• Let’s look at
the most simplistic case. What’s weird about it?
• Let’s Fix that
10 Feb. 2020 DTU Electrical Engineering 4

Image Formation
• Let’s look at
the most simplistic case. What’s weird about it?
• Let’s Fix that
• Let’s get some notation
amera rigin
10 Feb. 2020 DTU Electrical Engineering 4

Camera Extrinsics and Instrisics
• The orientation of the camera wrt Word coordinates is called extrinsics
• The rest of Parameters are called intrinsics
• We will see how points in 3D are projected on the sensor
amera rigin
10 Feb. 2020 DTU Electrical Engineering 8

Camera Extrinsics
• We need to express a 3D point P
𝑥𝑦𝑧𝑇 𝒙𝑝=𝑝 𝑝 𝑝
in the camera coordinates
amera rigin
𝑘𝒙 = 𝑘𝑥 𝑘𝑦 𝑘𝑧
• To do so we need to use the camera Origin
𝑘𝒙 = 𝑘𝑥 𝑘𝑦 𝑘𝑧
𝒙 𝑤𝑹𝑐 𝑤𝒕𝑐 𝑘𝒙 𝒑=𝒑
1 𝟎1𝑥3 1 1
𝑘𝒙 𝑤𝑹𝑇 −𝑤𝑹𝑇𝑤𝒕 𝒙 𝑘𝒙
֜𝒑=𝑐 𝑐𝑐𝒑֜𝒑=𝑹𝟎3𝑥13𝑥3 𝒑
𝑰 𝒕 𝒙 1 𝟎1𝑥3 1 1 1 𝟎1𝑥3 1 𝟎1𝑥3 1 1
10 Feb. 2020 DTU Electrical Engineering 9

Camera Extrinsics
• So now we have a description for the 3D point in camera frame
𝑘𝑥 𝑘𝑦 𝑘𝑧 𝑝 𝑝 𝑝
1𝑇 = 𝑹3𝑥3 𝟎3𝑥1 𝑰3𝑥3 𝒕3𝑥1 𝟎1𝑥3 1 𝟎1𝑥3 1
• To get to the image frame we will have to drop a dimension
rotation translation Homogeneous Point In World Coordinates
10 Feb. 2020 DTU Electrical Engineering 10

Projection
• Let’s get rid of things we don’t need
amera rigin
10 Feb. 2020 DTU Electrical Engineering 11

Projection
• Let’s get rid of things we don’t need
10 Feb. 2020 DTU Electrical Engineering 11

Projection
• Let’s get rid of things we don’t need
• So it is clear that the position of the projection on the image frame is dependent on the ratio x/z
10 Feb. 2020 DTU Electrical Engineering 11

Projection
• Let’s get rid of things we don’t need
• So it is clear that the position of the projection on the image frame is dependent on the ratio x/z
• In fact cxp= -f * kxp/ kzp and cyp= -f * kyp/ kzp
10 Feb. 2020 DTU Electrical Engineering 11

Projection
• From the previous
𝑘𝑥 𝑘𝑦 𝑘𝑧 𝑝 𝑝 𝑝
• We get to the following
s s T [x,y,w]=
1𝑇 = 𝑹3𝑥3 𝟎3𝑥3 𝑰3𝑥3 𝒕3𝑥1 𝟎1𝑥3 1 𝟎1𝑥3 1
rotation translation Homogeneous Point In World Coordinates
𝑹 𝒕 𝑥 𝑦 𝑧 1𝑇 3𝑥3 3𝑥1 𝑝 𝑝 𝑝
Intrinsics
• This is the analytic version of the projection matrix
Homogeneous Point
in World Coordinates
• Where W is the scale
Camera Matrix
exttrinsics
10 Feb. 2020 DTU Electrical Engineering 17

Projection Matrix
• We can define the Projection Matrix as follows
• This is great news for Calibration! sx • It means that our problem is a system of linear equations
• How many unknowns?
• 11: (5 + 6) Intrinsic+Extrinsic
Notation Disclaimer: In this slide X Y Z refer to 3D world points (not matrices) and x,y to 2D camera points
10 Feb. 2020 DTU Electrical Engineering 18

Calibration – Known 3D positions
• We defined our camera projection as follows:
• f we “sample” a set of points in 3D and 2D: [ , , ] → [x,y] we get the following analytic equations:
• However, λ is used for scale and therefore does not provide a 3rd equation
Notation Disclaimer: In this slide X Y Z refer to 3D world points (not matrices) and x,y to 2D camera points
10 Feb. 2020 DTU Electrical Engineering 19

Calibration – Known 3D positions
• So for every 3D to 2D correspondence we get 2 equations:
• How many points do we need? • 6 Points x 2 equations
Notation Disclaimer: In this slide X Y Z refer to 3D world points (not matrices) and x,y to 2D camera points
10 Feb. 2020 DTU Electrical Engineering 20

• For every point we get the following
0 = 𝑝11𝑋 + 𝑝12𝑌 + 𝑝13𝑍 + 𝑝 − 𝑝31𝑥𝑋 − 𝑝32𝑥𝑌 − 𝑝33𝑥𝑍 − 𝑝34𝑥 0=𝑝21𝑋+𝑝22𝑌+𝑝23𝑍+𝑝24 −𝑝31𝑦𝑋−𝑝32𝑦𝑌−𝑝33𝑦𝑍−𝑝34𝑦
• We can format all the linear equations in a single matrix: Ap=0
𝑝11 𝑝12 𝑝13 𝑝14
0 0 0 0 𝑋 𝑌 𝑍 1 −𝑦𝑋 −𝑦𝑌 −𝑦𝑍 −𝑦 111 1111111𝑝22
𝑋 𝑌 𝑍 1 0 0 0 0 −𝑥𝑋 −𝑥𝑌 −𝑥𝑍 −𝑥
1 1 1 11 11 11 1𝑝21
𝑋 𝑌 𝑍 1 0 0 0 0 −𝑥 𝑋 −𝑥 𝑌 −𝑥 𝑍 −𝑥
𝑛 𝑛 𝑛 𝑛𝑛 𝑛𝑛 𝑛𝑛 𝑛𝑝24
0 0 0 0 𝑋 𝑌 𝑍 1 −𝑦𝑋 −𝑦𝑌 −𝑥𝑍 −𝑥 𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛
𝑝31 𝑝32 𝑝33 𝑝34
Notation Disclaimer: In this slide X Y Z refer to 3D world points (not matrices) and x,y to 2D camera points
10 Feb. 2020 DTU Electrical Engineering 21

Calibration – Known 3D positions
• The solution is given by the following process which is itself called DLT: – Calculate the Singular Value decomposition (SVD)
SVD : A = UDVT
– Get the last column of V
P = Vsmallest (column of V corr. to smallest singular value)
-Reshape into the P matrix
10 Feb. 2020 DTU Electrical Engineering 22

Calibration – Known 3D positions
• To summarize:
– Get min 6 3D – 2D correspondences
– Form matrix A (AP=0)
– Calculate SVD
– Get Last column of V which corresponds to the values of P
10 Feb. 2020 DTU Electrical Engineering 23

Calibration in real world
• In Real World we do not have 3D points.
• Or do we?
• What if we could use some sort of predefined setup which would allow us to know the relative position of 3D points?
• It turns out that if we have a calibration pattern we can know their relative positions of all points as they are on a known configuration
10 Feb. 2020 DTU Electrical Engineering 24

Calibration using homography
• Since Z = 0 we lose one degree of freedom:
• We can then redefine our P as:
• This is called homography and allows us to project points from one plane to another
Notation Disclaimer: In this slide X Y Z refer to 3D world points (not matrices) and x,y to 2D camera points
10 Feb. 2020 DTU Electrical Engineering 25

Calibration using homography
• How to calculate the homography:
𝑤′𝑥′ 𝐱′= 𝑤′𝑦′
h1 𝐇= h4 h7
h2 h3 h5 h6 h8 h9
• The homography has 9 elements. However, similar to before due to the scale there are only 8 unkowns • Therefore, we need minimum 4 correspondences to setup the problem as a DLT
−𝑥 −𝑦 −1 0 0 0 𝑥𝑥′ 𝑥𝑥′ 𝑥′ h4
0 0 0 −𝑥 −𝑦 −1 𝑦𝑥′ 𝑦𝑦′ 𝑦′
10 Feb. 2020 DTU Electrical Engineering 26

Calibration using homography
• With a static camera move the pattern and get images.
– Make sure to not present only one orientation, as there will be a problem with the estimation of the homography
• Calculate correspondences using the pattern • Solve using Normalized DLT
– Normalize coordinates for each image
• Translate for zero mean
• Scale so that average distance to origin is ~sqrt(2)
– Form Matrix A
– Solve using SVD • Calculate SVD
• Use last column of V to get the homography.
– Denormalize
10 Feb. 2020 DTU Electrical Engineering 27

Calibration using homography
• However the DLT solution is known to be prone to outliers. • Other approaches exist to solving the calibration:
– Nonlinear least squares
– RANSAC • RANSAC:
1. Choose number of samples N
2. Choose 4 random potential matches
3. Compute H using normalized DLT
4. Project points from x to x’ for ea h potentiall
5. Count points with projected distance < t 6. Repeat steps 2-5 N times Choose H with most inliers mat hing pair: 10 Feb. 2020 DTU Electrical Engineering 28 Calibration - Distortion • Ok, but still, What about the lens Distortion? Barrel Pincushion Fisheye 10 Feb. 2020 DTU Electrical Engineering 29 Calibration - Distortion • So how do we model it: 10 Feb. 2020 DTU Electrical Engineering 30 Calibration - Distortion • So how do we model it: • Usually a quartic (biquadratic) polynomial is used: • OR higher order 10 Feb. 2020 DTU Electrical Engineering 30 Calibration - Distortion • So how do we model it: • Usually a quartic (biquadratic) polynomial is used: • OR higher order Undistorted 10 Feb. 2020 DTU Electrical Engineering 30 Calibration - Distortion • How to calibrate for the distortion coefficients: – First do DLT for the Projection Matrix – Then form a non linear least square problem that includes both the linear projection and the non-linear distortion using the algorithm. 10 Feb. 2020 DTU Electrical Engineering 33 Conclusion • What did we learn? 10 Feb. 2020 DTU Electrical Engineering 35 Perception for Autonomous Systems 31392: Camera Matrix and Camera Calibration Lecturer: —PhD 10 Feb. 2020 DTU Electrical Engineering 36 程序代写 CS代考加微信: powcoder QQ: 1823890830 Email: powcoder@163.com

Related Posts