EE 5806: Topics in Image Processing
Bernard Chiu
bcychiu@cityu.edu.hk
1
Textbook
• Gonzalez and Woods, “Digital image processing,” 4th edition, Pearson Prentice Hall, 2018.
2
Course assessment
• 2-hour examination: 50%
• ContinuousAssessment:50%
– 2 assignments: 7% each – 2 tests: 13% each
– Group project: 10%
• To pass the course:
– At least 30% in the examination
– At least 30% in continuous assessment
3
Assignments and exam schedule
• 2Assignments
– Due Week 5 Friday: Oct. 2, 2020
– Due Week 12 Friday: Nov. 20, 2020
• 2 Tests (1 hour each)
– Week 6 in lecture: Oct. 7, 2020
– Week 11 in lecture: Nov. 11, 2020
4
Policy for Late Assignment Submission
• 50% penalty per day
• If the assignment is late for 1 day, we will multiply the raw assignment score by 0.5.
• A zero score will be given if the assignment is late for 2 days or more than 2 days.
5
Group Project
• Proposeprojectsandformgroups
– Form groups of no more than 5 students. Sign up for a group by yourself in Canvas.
– Submit a project proposal by Week 4 Friday (Sept. 25, 2020)
– Proposal should be 3-4 pages and includes motivation, approach and a description of data in single-column and single-space format.
• Oral Presentation
– A 15-minute presentation with demonstration on Week 12 or 13
• FinalReport
– A final report of the project with 8 to 12 pages in in single-column and single-space format.
– Deadline: Week 14 Friday (Dec. 4, 2020)
6
Possible Project Directions
• Something you are interested in or that may help your work or research
• The project does not need to be original, but the topic should
– Involve practical applications or focus on solving a practical issue.
– Be sufficiently motivated. Why do you choose to work on this technique? Is it better than other methods?
• Could investigate something we have covered briefly, or do not have time to discuss in more detail.
7
Some ideas
• Investigate two feature detection algorithms; explain how they work, compare their performance, why one works better than the other.
• Apply an algorithm to a new problem domain, evaluate its performance
• The textbook has many references to “further reading”
8
Example Projects
• Prostate segmentation or lesion identification from the ProstateX dataset
https://wiki.cancerimagingarchive.net/display/Public/S PIE-AAPM-NCI+PROSTATEx+Challenges
9
CEF
As EE5806 is a course supported by Continuing Education Fund (CEF), local student can apply reimbursement for the tuition fee of this course if he/she attains a minimum attendance rate 75% and get at least 50% in this course.
10
Digital image processing
• Definition: Transform one image into another image to achieve some goals.
• We will consider the following tasks as the course proceeds:
– Image enhancement
– Image restoration and reconstruction – Image segmentation
– Image coding and compression
11
Image Acquisition and Display
12
Image Enhancement
• Image enhancement is the process of adjusting digital images so that the results are more suitable for display or further processing
– Contrast enhancement – Noise removal
– Deburring
13
Image Restoration
• To restore a degraded image to its original form. Image degradation may be attributable to:
– Optical blur
– Motion blur
– Spatial quantization
– Additive intensity noise
14
Image Restoration: Inpainting
• Inpainting: Automated texture generation to fill in missing regions.
15
Image Segmentation
• Image segmentation is the partitioning of an image into different regions. This is done to extract important information of objects in the image (e.g., size, location)
16
Image Compression
• Image compression is done to reduce the cost of storage and transmission of digital images.
17
1.1 Image Sensing and Acquisition
18
Image Sensors
• Image sensors contain millions of photosensitive diodes known as photosites.
• Photosites are individual picture elements (“pixels”), which are arranged in an array to capture images.
• The brightness recorded by each photosite is stored as a set of digital numbers.
19
Types of Image Sensors
20
Operation of CCD
1. Charge accumulates during exposure
2. Collected charges shifted out by horizontal and then vertical shift registers
3. Each pixel is converted to the voltage
4. Voltage is amplified by an amplifier
21
Blooming in CCD
• The charge collected by a pixel leaks to other pixels.
• Electrons move more easily in the vertical direction, resulting in a vertical streak.
22
Operation of CMOS
• In a CMOS sensor, each pixel has its own charge- to-voltage conversion, amplifiers and digitization circuitsNo blooming, faster operation.
• Digital values are read in
a line-by-line fashion
Rolling shutter artifact
• CMOS circuitry dissipates less power.
Rolling_shutter_SMIL.svg
23
Comparison of CCD and CMOS
CCD
CMOS
Broadcast cameras (still industry standard for TV)
Phone, Web, and digital SLR cameras
Better image quality
Cheaper and faster (data transfer)
Susceptible to the blooming effect
Susceptible to the rolling shutter effect
24
Colour Capturing in CCD
• 3 CCDs capturing red, green and blue images.
• Aprismisusedto separate the original image into red, green and blue images.
25
Colour Capturing in CMOS
• Arranging colour filters on a square grid of photosensors.
• Bayer pattern – 50% green
– 25% red – 25% blue
26
Demosiacing
Reconstruction by summing the three channels
27
Demosiacing
Bayer Filter Samples
Source Image
Red Green Blue
Reconstructed Image
28
1.2 Digital Imaging Fundamentals
An image formation model
• Components – Illumination – Reflection
𝑓𝑥,𝑦 𝑖𝑥,𝑦𝑟𝑥,𝑦 0𝑖𝑥,𝑦 ∞ 0𝑟𝑥,𝑦 1
30
Definition of digital image
• Matrix of numbers that can be displayed.
31
Digital image
• Pixel (short for picture element)
– Element in image matrix
– Identified by a pair of indices indicating the row and column where the pixel is located (e.g., Pixel (1,2) is shaded)
• Greyscale
– The value associated with a pixel that represents the brightness at that location.
– The greyscale value at Pixel (1,2) is 122.
– Generally integer values, ranging from 0 to a maximum value.
• Colour image
– 3 functions can be defined for the red, green, blue (RGB) channels
32
Number of grey levels
• The number of grey levels (L) is usually a power of 2:
L = 2B
where B is the number of bits used to
represent the grey level at each pixel.
• e.g.,B=8impliesL=28=256greylevels ranging from 0 to 255.
• Checkpoint: What does B = 1 imply?
33
Amount of Image Data
For an MxN image:
• Number of bits per pixel per channel: k bits
• Numberofgraylevelsperpixelperchannel:
𝒌 levels
• Pixelvaluedynamicrange:[0,L-1]
• Numberofbitsrequiredtostoreangray-scale image: b = MNk bits
• Number of bytes per image: B = MNk/8 Bytes
• NumberofbyteswithcoluorimagewithRGB
component: B = 3*MNk/8
• Example:a8-bit gray-scaleimage – bytes = 2.0736 MB
34
Digital image representation
• Adigitalimageis denoted by I(i,j)
– i = row number (0, 1, …, M-1)
– j = column number (0, 1, …, N-1)
– I(i,j) = gray level at row i, column j
– Note: capital I for image; lowercase i for row number
35
Image display
• Colourmap
– Table that specifies how each greyscale value is mapped into a shade of grey.
– e.g., for L = 256 gray levels, 0 maps to black and 255 maps to white. Values between these two extremes map linearly into different levels of grey.
Display
Colourmap
36
Image digitization
• Any visual scene can be represented by a 2D continuous function I(x,y).
• I(x,y) is the brightness or intensity of a point at location (x, y), where I, x and y take on a continuum of values.
• A continuous analog representation cannot be conveniently interpreted by a computer.
• We need to digitalize analog images.
37
Image digitalization
• Digitalization: Conversion of a continuous- tone image (i.e., I(x,y)) to a digital image (i.e., I(i,j)).
• Digitalization consists of two steps: – Sampling
– Quantization
38
Sampling
• The continuous image I(x,y) is scanned, and its brightness is measured and sampled at discrete locations to form an element in a rectangular matrix.
• Note that the output image is NOT a digital image yet.
39
Quantization
• Each element in the sampled image is quantized (i.e., replaced by an integer).
• Quantized values are called grey levels.
• A B-bit image has 2B grey levels, ranging from 0 to 2B-1.
• The range of grey levels is called the dynamic range of
the image.
40
Spatial resolution
• Spatial resolution: Physical size of a pixel.
• What happens when the resolution decreases (i.e., pixel size increases)?
The picture at this resolution looks blocky, demonstrating an artifact called pixelation.
41
Brightness resolution
• Thebrightnessateachpointinacontinuous-tone image was replaced by an integer greyscale value during quantization.
• Brightnessresolutionreferstohowaccuratethe greyscale value can represent the intensity of the corresponding point in the original analog scene.
42
Brightness resolution
• The number of bits available affects how accurately a continuous-tone image is represented.
• e.g., The figures below show the same continuous-tone image quantized with 8 and 3 bits respectively.
This image is continuous
This image is not continuous (brightness contouring)
43
Brightness resolution
• The same x-ray image quantized with 8 bits to 1 bit.
44
Brightness resolution
• With fewer bits, a wider range of intensities in the original continuous-tone image is mapped into a single grey level. Compare Δ2 and Δ8 below.
• Gradual variations of intensity cannot be accurately represented with small number of bits.
• More bitsBetter brightness resolution.
45
Image Interpolation
• We often need to estimate (interpolate) the value of an image at non-integer locations.
• Example applications:
– Zooming, shrinking, rotate and applying other geometric corrections.
46
Application: Resizing an Image
47
Image Interpolation Methods
• Nearest neighbour
– Gray level of (i,j) is assigned to be that of its nearest neighbour.
• Bilinear
– Weighted average of 4
closest neighbours.
• Bicubic
– Weighted average of 4
closest neighbours.
– Most expensive computationally, but gives best results.
48
Nearest Neighbour
• Colour are defined in black points in the figure.
• Locations inside each colour cell have the black point in the cell centre as the nearest neighbour.
49
Bilinear interpolation
• Forbilinearinterpolation,wedefinealocal coordinate system (s,t) with Neighbour 1 as the origin.
• sandtarebetween0and1asshownbelow.
• Point(i,j),thepointwewantagraylevel,hasa local coordinate (s,t).
50
1D linear interpolation
• Before describing bilinear interpolation, let’s recall how we do 1D linear interpolation.
• We have a 1D function with data value available only when x is an integer.
• Want to get f(t) (t is not an integer) by linear interpolation.
51
Bilinear interpolation
• Now, let the gray level at Neighbours 1, 2, 3 and 4 respectively are p(0,0), p(0,1), p(1,0) and p(1,1).
• We want gray level at (s,t), denoted by p(s,t).
• Let’s interpolate horizontally first just like what we did in the 1D case to obtain gray level at:
– (0,t)denotedbypU. – (1,t)denotedbypL.
• Then, we interpolate vertically to get p(s,t):
52
Bicubic Interpolation
• Theinterpolatedvalueisa
weighted sum of the 4×4
neighbourhood:
• Computationofthe16 coefficients involves writing 16 equations based on the values and their derivatives at the four corners. See here for details:
https://en.wikipedia.org/wiki/Bicu bic_interpolation
53
Interpolation Example 1
54
Interpolation Example 2
55
1.3 Python-based Image Processing Tools
56
Image Processing Tools
• Python 3 as the programming language
• Anaconda as the software tool
– Includes many pre-installed python libraries – numpy, matplotLib, skimage, etc
• Jupyter Lab as a development environment
• OpenCV (Open Computer Vision Library) is a computer vision library for real-time image processing
57
Install Anaconda
• Anaconda Python Package
– https://www.anaconda.com/products/individual
58
JupyterLab
• Most programming can be done in a JupyterLab notebook.
Current Directory
Cell: Code can be typed in a cell and then execute by typing Shift- Enter
59
Running a Python Script
• Instead of typing in commands one at a time in the cell, users can type commands to a .py file and automate a sequence of actions in the Jupyter notebook.
• Example:
– Type command in a .py file. Save it as DemoResize.py
– Type %run DemoResize in one of the cell in Jupyter notebook
– Press Shift+Enter to run cell
60
Programming with Python Functions
• Function: a piece of code that accomplishes a certain task and optionally returns a value.
• A function consists of the following components:
1. Keyword def that marks the start of the function header.
• Import and call the function in Jupyter notebook:
2. A function name to uniquely identify the function.
3. One or more valid python statements that make up the function body. Statements must have the same indentation level (usually 4 spaces).
4. An optional return statement to return a value from the function.
61
More Python
• More Python in Week 2.
• Try to install Anaconda in your computer.
• Try to get familiar with the materials in the folder Demo2, which covers an introduction to:
– Python language – OpenCV library – skimage
62
1.4 Colour Fundamentals
63
Colour Fundamentals
• White light can be decomposed into a continuous light spectrum ranging from red to violet.
64
Colour Perception
• Cones are sensors in the eye responsible for colour vision.
• 6 to 7 million cones in the eye can be divided into 3 types of cones:
– L => sensitive to RED (65%)
– M => sensitive to GREEN (33%)
– S => sensitive to BLUE (2%). Most sensitive among the three types.
• The human eye combines 3 primary colors detected by these three types cones to discern all visible colors. This suggests that color can be represented in a 3D space.
65
Colour Model and Colour Space
• Colourmodel
– An abstract mathematical model describing the way colours can be represented as tuples of numbers.
Gamut of Colour Monitors
– Limited in the range of colors they can represent and cannot represent all colors in the visible spectrum.
• ColourSpace(Gamut)
– The range of colour covered by a colour model.
Gamut of Printing Devices
66
RGB Colour Model
• Additive colour model.
• Generates colours by adding the three primary colours: red, green and blue.
• Used in computer monitor, LCD screens, projectors
• RGB Color Model
– CIE Standard Primary Colours • Red: 700 nm
• Green: 546 nm • Blue: 436 nm
67
RGB Colour Model
• Colour is defined by a 3-tuple that gives the amount of red, green and blue light in the colour.
• R, G, B values range [0 1]
• Grayscale along the diagonal (R = B = G)
• If each component is quantized into 256 levels, the total number of different colors that can be produced is:
2 2 16,777,216
colours.
• RGB safe colour:
– Quantize each component into 6 levels
24-bit RGB colour cube
RGB safe-colour cube
68
RBG Image Generation
69
CMY Model
• Representthepigmentusedto generate colours.
• Example:Whenwhitelightis reflected off a coating of cyan pigment, cyan is displayed and the red light is absorbed.
• ThethreecoordinatesoftheCMYmodelcan be interpreted as:
• Subtractionmodel
=
70
CMY Color Model
𝐶1
• 𝑀 0Green
𝑌1
– Red and Blue are totally absorbed
𝐶1
• 𝑀 1Blue
𝑌0
– Red and Green are totally absorbed
𝐶0
• 𝑀 1Red
𝑌1
– Green and Blue are totally absorbed
𝐶1
• 𝑀 1Black
𝑌1
– Red, Green and Blue are totally absorbed
71
CMYK Model
• Why CMYK?
– Equal amounts of the pigment primaries, cyan, magenta, and yellow should produce black. In practice, combining these colors for printing produces a muddy- looking brownish black.
– To produce true black, the predominant color in printing, the fourth color, black, is added, giving rise to the CMYK color model.
72
Coversion from CMY to CMYK
• Givenacolorinnormalized CMY space as
• Kisameasureofthe ‘blackness’ of the color and essentially serves as an offset, after which the remaining amounts of cyan, magenta and yellow are ‘added’
73
CMY vs. CMYK
74
Colour and Brightness
• Any color can be decomposed into
– A brightness component. This corresponds to the ‘grayscale’ version of the color
– A color component. All other information is ‘colour’ or ‘chroma’
• In RGB and CMY space, the brightness and chroma are distributed over each of the three components
• HSV colour mode separate chroma and brightness
75
HSV Colour Model
• Colors are distinguished from one another based on hue, saturation and brightness.
• Hue
– The dominant color
• Saturation
– Strength of the color.
• Brightness (Intensity or Value) – How much light there is.
• Hue + Saturation define the chromaticity of
a color.
76
HSV Colour Model
• The HSV color space is a cylinder and hence HSV coordinates are given with cylindrical coordinates.
– Hue is an angular measure [0-360] that is often normalized to be [0-1] where 1 correspond to 360 degrees.
– Saturation range: [0, 1]
– Brightness range: [0, 1]
• Fully de-saturated colors are different shades of gray (no color)
77
Comments on HSV
• The center of the cylinder is the grayscale
– Black (0, 0, 0) to White (0, 0, 1)
• Red is chosen as the reference for hue (i.e., Hue = 0 for red)
• If intensity is zero then what do H and S mean?
• HSV is useful since when filtering an image we often need to process only the brightness band. Using HSV space we can process only brightness without altering the color.
78
Colour Coordinate Transformation
79
Colour Image Processing in Python
80
Transform to HSV
81
Summary
– KnowhowimagesareacquiredbyCCDandCMOSsensors.
– KnowprosandconsofCCDandCMOSsensors.
– KnowhowcolourimagesareacquiredbyCCDandCMOS Sensors.
– Knowthefollowingtermsrelatingtodigitalization • Sampling
• Quantization
• Dynamic range
• Spatial resolution – Pixelation
• Brightness resolution – Brightness contouring
– Know the amount of storage required for an image.
– Knowhownearestneighborandbilinearinterpolationworks.
– KnowhowtoinstallAnacondaandhowtouseJupyterLab.
– Knowdifferentcolourmodelsintroducedandwhattheyare good for.
82