Chapter 6 Geometric Operations and Image Registration
Definition
• Geometricoperationsareoperationsthatmodify spatial relationship between pixels of an image or change the shape of objects within an image.
• e.g.,Movingaroundobjectsinanimage.(a) Original image, (b) rotate 450 around centre and (c) Translate 17 pixels right and 23 pixels down.
(a) (b) (c)
2
Mathematical definition
a and b are functions that specify the geometric operation.
3
Linear operations
• Translation
• Scaling
• Rotation
• We only consider linear operations in this course.
4
Translation
5
Translation
Original image
6
Scaling about the origin
7
Scaling about the origin
si=sj=1 si=sj=0.5 si=sj=1.5
8
Scaling about a point
• Scaletheimageaboutaspecifiedpoint(i0,j0)(i.e., scale the distance between a pixel (i,j) and a specified pixel (i0,j0).)
9
Scaling about a point
si=sj=0.5
about the centre of the image
10
Rotation about origin
• Howtoexpress(i’,j’)?
Property of linear geometric operation
• Let T be a linear geometric operation, and x1 and x2 be two arbitrary points.
• Ifx1→T(x1)andx2→T(x2), then for any constants a and b,
• ax1+bx2 → T(ax1+bx2) = aT(x1) + bT(x2)
11
Rotation about origin
• Let’s denote the geometric op required to rotate about the origin by T.
• We need ← not easy to get. • However, easy to get
12
Rotation about origin
• Based on the property of linear geom. op.
13
Rotation about origin
14
Rotation about a point
• How about if we rotate about (i0, j0) instead of the origin?
• We temporarily treat (i0, j0) as the origin in a new coordinate system. Point (i, j) in the original coordinate system becomes Point (i-i0, j-j0) in this new coordinate system.
• The rotated point has the following coordinate in the new coordinate system:
• The rotated point has the following coordinate the original coordinate system:
15
Rotation about a point
16
Rotation about a point
Rotation about image centre for 45 degrees counter-clockwise
17
Gray-level interpolation
• Problem: Transformed coordinate may not be integer. Output image is indexed by integer coordinate.
• Solution: Gray-level interpolation
• Gray-level interpolation: determine gray-level at image coordinate, given gray-level of transformed pixels.
18
Gray-level interpolation
• First approach: Forward mapping
• Compute new coordinates
• Use an interpolation scheme to compute the gray levels of four neighbouring pixels.
19
Gray-level interpolation
• Problems with forward mapping:
– Leave holes in output: Some pixels do not have an associated gray level value.
– For some pixels, gray levels are computed multiple times. Computational power is wasted.
20
Gray-level interpolation
• Secondapproach:Backwardmapping
– Visit each pixel (i’,j’) with integer coordinate in I0.
– Apply inverse mapping to map from (i’,j’) to (i,j).
– Use interpolation to find gray level at (i,j) based on the gray levels of 4 neighbours.
– no holes; gray level of each output pixel is determined based on one interpolation between 4 pixels in input image.
21
Interpolation methods
• We will describe two approaches in this course: (i) nearest neighbor (ii) bilinear
• Nearest neighbor: Gray level of (i,j) is assigned to be that of its nearest neighbour.
• (i,j)inIhas4 neighbours.
• Neighbour 1 is closest.
• Take gray value of Neighbour 1.
22
Bilinear interpolation
• Forbilinearinterpolation,wedefinealocal coordinate system (s,t) with Neighbour 1 as the origin.
• sandtarebetween0and1asshownbelow.
• Point(i,j),thepointwewantagraylevel,hasa local coordinate (s,t).
23
Bilinear interpolation
• We are given the output pixel coordinates (i’,j’) and the image transform T (from which input pixel coordinate (i,j) can be obtained).
• But how do we know the local coordinate (s,t) given the global coordinate of pixel is (i,j)?
24
Bilinear interpolation
Neighbour
Global Coor.
Local Coor.
1 2 3 4 Self
(floor(i), floor(j)) (floor(i), ceil(j)) (ceil(i), floor(j)) (ceil(i), ceil(j)) (i,j)
(0, 0)
(0, 1)
(1, 0)
(1, 1) (i-floor(i), j-floor(j))
25
Interpolation methods
• Before describing bilinear interpolation, let’s recall how we do 1D linear interpolation.
• We have a 1D function with data value available only when x is an integer.
• Want to get f(t) (t is not an integer) by linear interpolation.
26
Bilinear interpolation
• Now, let the gray level at Neighbours 1, 2, 3 and 4 respectively are p(0,0), p(0,1), p(1,0) and p(1,1).
• We want gray level at (s,t), denoted by p(s,t).
• Let’s interpolate horizontally first just like what we did in the 1D case to obtain gray level at:
– (0,t) denoted by pU. – (1,t) denoted by pL.
• Then, we interpolate vertically to get p(s,t):
27
Gray level interpolation: Example 1
• Given the gray levels of its 4 neighbouring pixels displayed in blue, calculate the gray level at location (i, j) = (234.3, 101.7) using:
(a) Nearest neighbour interpolation (b) Bilinear interpolation
28
Gray level interpolation: Example 1
(a) Nearest neighbour is (234, 102). Gray level = 45.
(b) Bilinear interpolation:
–
(i = 234.3, j = 101.7) is associated with a local coordinate of (s, t) where:
–
Refer to p. 17,
• s=i–floor(i)=234.3–234=0.3 • t=j–floor(j)=101.7–101=0.7
29
Gray level interpolation: Example 1
• Now (s, t) = (0.3, 0.7)
1st term 2nd term 3rd term 4th term
0.7 0.3 0.7 0.7 0.3 0.3 0.3 0.7
18 3.78 45 22.05 52 4.68 36 7.56
• Gray level at (234.3, 101.7) is 38.07 ~ 38
Sum = 38.07
30
Gray level interpolation: Example 2
• A rotational operation is applied to image I:
• Given the following gray levels for input image, find the gray level of the pixel (i’, j’) = (378, 120) in the output image using (a) nearest neighbour and (b) bilinear interpolation.
(i,j)
I(i,j)
(182, 352) 18 (182, 353) 45 (183, 352) 52 (183, 353) 36
31
Gray level interpolation: Example 2
• Note that R-1 = RT for rotational matrices
• See example 1 for calculating gray level using interpolation
32
Bilinear vs. Nearest neighbour
Original
Nearest neighbour
Bilinear
33
Python implementation
import numpy as np
import math
import cv2
import matplotlib.pyplot as plt
def rotateimage(im, theta):
“””
rotateimage: rotate im about its center.
:param im: the original image
:param theta: the rotation angle in degree
:return: im2: the rotated image
“””
# get image center
rows, cols = im.shape
ic = np.round((rows-1)/2)
jc = np.round((cols-1)/2)
# convert theta to radians
theta = -theta * math.pi / 180
im2 = np.zeros([rows, cols])
for jprime in range(cols):
for iprime in range(rows):
i = math.cos(theta)*(iprime-ic)-math.sin(theta)*(jprime-jc)+ic j = math.sin(theta)*(iprime-ic)+math.cos(theta)*(jprime-jc)+jc im2[iprime, jprime] = nearestneighbour(im, i, j)
return im2
34
Python implementation
def nearestneighbour(im, i, j):
rows, cols = im.shape
iround = int(np.round(i))
jround = int(np.round(j))
if iround < 0 or iround >= rows:
# out of bound
return 0
elif jround < 0 or jround >= cols:
# out of bound
return 0
else:
# within bound
return im[iround, jround]
35
2D transformation using Python
• For details, read Geometric Transformations of Images in OpenCV-Python Tutorial
• https://opencv-python- tutroals.readthedocs.io/en/latest/py_tutorials /py_imgproc/py_geometric_transformations/ py_geometric_transformations.htmls
36
2D transformation using built-in tool
2 steps:
Overview
1. Define parameters (transformation matrix) for spatial transformation.
Input Image
Transformation matrix
2. Perform the transformation by passing the input image and the transformation matrix to cv2.warpAffine().
cv2.warpAffine
Transformed image
37
Step 1
• All transformation we dealt with can be expressed as an affine transformation.
• The affine transformation can be written in the matrix form as:
as follows:
=
=
100111
• Python expresses the same relationship. We just need to create the transformation matrix
=
38
Step 1
Operation
Coordinate equations
*Affine matrix, T
Translation
′=+ ′ =+
1 0 0 1 001
Scaling about origin
′= ′=
0 0
0 0 001
0 0 −0 0 0−0
Scaling about apoint( , )
′= +(−)=+( −) 0000
Rotation about origin for clockwise
′ = − ′ = +
− 0 0
Rotation about apoint( , )
0 0 0 ′ =(− )+(− )+
−
00
001
00
000001
′ =0 +(−0)=+(0 −0)
′ =(−0)−(−0)+0
0 0 1
*Note that only the first two rows of T are needed in Python.
39
=−+( − + )
=++( − − )
0 0 0
Step 2
• Step 2: Perform the transformation.
im2 = cv2.warpAffine(im, T, (width, height)) – im is the input image; im2 is the output
image
– T is the transformation matrix. We can create it as a Numpy array of type np.float32
– width and height specify the size of the output image
40
Putting it all together …
import numpy as np
import math
import cv2
import matplotlib.pyplot as plt
Rotate about
centre of image counterclockwise for 450
im = cv2.imread(‘pout.tif’, cv2.IMREAD_UNCHANGED) plt.imshow(im, cmap=’gray’, vmin = 0, vmax = 255) plt.axis(‘off’)
plt.show()
height, width = im.shape
# Compute centre of image
i0 = np.round(0.5*(width-1)) j0 = np.round(0.5*(height-1))
# Step 1: Define affine matrix, T
theta = 45 # angle
theta = math.radians(-1*theta) # radian
tx = i0-i0*math.cos(theta)+j0*math.sin(theta) ty = j0-i0*math.sin(theta)-j0*math.cos(theta) T = np.float32([[math.cos(theta), – 1*math.sin(theta), tx], [math.sin(theta), math.cos(theta), ty]])
# Step 2: Perform the transformation
im2 = cv2.warpAffine(im, T, (width, height)) plt.imshow(im2, cmap=’gray’, vmin = 0, vmax = 255) plt.axis(‘off’)
plt.show()
41
Putting it all together …
import numpy as np
import math
import cv2
import matplotlib.pyplot as plt
• We can also create the transformation matrix in
im = cv2.imread(‘pout.tif’, cv2.IMREAD_UNCHANGED) plt.imshow(im, cmap=’gray’, vmin = 0, vmax = 255) plt.axis(‘off’)
plt.show()
Step 1 with the build-in function
height, width = im.shape
cv2.getRotationMa
# Compute centre of image
i0 = np.round(0.5*(width-1)) j0 = np.round(0.5*(height-1))
trix2D()
# Step 1: Define affine matrix, T
theta = 45 # angle
theta = math.radians(-1*theta) # radian
• T= cv2.getRotationMatrix2D (center=(i0, j0), angle=45, scale=1)
tx = i0-i0*math.cos(theta)+j0*math.sin(theta) ty = j0-i0*math.sin(theta)-j0*math.cos(theta) T = np.float32([[math.cos(theta), – 1*math.sin(theta), tx], [math.sin(theta), math.cos(theta), ty]])
# Step 2: Perform the transformation
im2 = cv2.warpAffine(im, T, (width, height)) plt.imshow(im2, cmap=’gray’, vmin = 0, vmax = 255) plt.axis(‘off’)
plt.show()
42
Summary
• Definition of geometric operation
• Linear spatial transformation – Translation
– Scaling about origin or an arbitrary point – Rotation about origin or an arbitrary point
• Definitions of forward and backward mapping. Why backward mapping is used instead of forward mapping?
• Gray-level interpolation – Nearest neighbour
– Bilinear
43
Summary
• Python implementation
• Using built-in function cv2.warpAffine for implementing geometric transformation.
• Landmark transformation
44