MULTIMEDIA RETRIEVAL
Semester 1, 2022
Content Based Retrieval I
Background
Copyright By PowCoder代写 加微信 powcoder
Visual feature extraction Color
Texture Shape
School of Computer Science
http://www.imdb.com/
Text vs. Image
Region Pair
School of Computer Science
Content-based Retrieval
Using visual features to reflect rich contents Color, shape, texture, …
Automatic feature extraction Plenty of applications
Query by example, query by sketch, …
No explicit semantic concepts
Difficult for indexing high dimensional features
A picture is worth a thousand words.
School of Computer Science
Content-based Retrieval
Content-based image retrieval diagram
School of Computer Science
Content-based Retrieval
Feature extraction is performed to obtain multi-dimensional feature vectors characterizing multimedia contents.
Different media has different information representations.
Appropriate similarity measurement is employed to measure the
similarity between query item and database item.
Feedback provides the interactions between users and systems.
Efficient indexing techniques are employed to organize databases.
Benchmarking is to evaluate retrieval performance. TREC (Text REtrieval Conference) ???
School of Computer Science
Audio Representation
Acoustic features
Subjective / Semantic features
School of Computer Science
Acoustic features
Loudness
The signal’s root-mean-square (RMS) level in decibels over a series of windowed frames.
Pitch is estimated by taking a series of short-time Fourier spectra.
Brightness
The centroid of the short-time Fourier magnitude spectra.
Bandwidth
The magnitude-weighted average of the differences between the spectral components and
the centroid. Harmony
It is computed by measuring the deviation of the sound’s line spectrum from a perfectly harmonic spectrum. It distinguishes between harmonic spectra, inharmonic spectra, and noise.
Refer to http://mpeg7.doc.gold.ac.uk/ for more MPEG-7 Audio Features
School of Computer Science
Acoustic features
Male Laughter
Feature Vectors:
1. Pitch [mean, variance, autocorrelation]
2. Amplitude [mean, variance, autocorrelation]
3. Brightness [mean, variance, autocorrelation]
4. Bandwidth [mean, variance, autocorrelation]
School of Computer Science
Audio Classification and Retrieval
Input Audio
Feature Extraction
Pattern Recognition
Documentary
School of Computer Science
SoundFisher
Textbook pp.108-112
SoundFisherTM is a new sound effects database management system featuring content-based recognition and retrieval. http://www.soundfisher.com/html/overview.html
School of Computer Science
Applications of Audio Classification and Retrieval
Audio database management Audio database browser
Audio editor
Assistance in video analysis
Surveillance, such as silence detection Sports
Movie genre classification
Speech recognition
Music genre classification Instrument identification
Textbook pp.96-106
School of Computer Science
Image Representation
Image content
Visual content
Semantic content
Obtained either by textual annotation or by complex inference procedures based on visual content
General visual content
Color, texture, shape, spatial relationship, etc.
Domain specific visual content
Application dependent and may involve domain knowledge
School of Computer Science
Image Representation
A visual content descriptor can be either global or local.
A global descriptor uses the visual features of the whole image.
A local descriptor uses the visual features of regions or objects to describe the image content, with the aid of region segmentation and object segmentation techniques.
School of Computer Science
Image Representation
Spatial relationship
Compression: fractal coding, JPEG
School of Computer Science
Human is more sensitive about color.
Color is very powerful in description and of
easy extraction.
Color varies considerably with the change of illumination, orientation of the surface, and the viewing geometry of the camera.
School of Computer Science
Color fundamentals
Color spaces
Color features
Color histogram
Color moments
Color coherence vector (CCV)
Similarity between colors
School of Computer Science
Color Fundamentals
Light and the Electromagnetic (EM) Spectrum
In 1666, Sir Isaac Newton discovered that when a beam of sunlight passes through a glass prism, the emerging beam of light is not white but consists instead of a continuous spectrum of colors ranging from violet at one end to red at the other.
The color spectrum may be divided into six broad regions: violet, blue, green, yellow, orange, and red.
R. Gonzalez, R. Woods, “Digital Image Processing”, , 2002.
School of Computer Science
Color Fundamentals
Light and the Electromagnetic (EM) Spectrum
Light is a form of electromagnetic radiation
The range of colors we perceive in visible light represents a very
small portion of the electromagnetic spectrum.
When view in full color, no color in the spectrum ends abruptly, but rather each color blends smoothly into the next.
Wavelengths comprising the visible range of the electromagnetic spectrum
R. Gonzalez, R. Woods, “Digital Image Processing”, , 2002.
School of Computer Science
Color Fundamentals
Light and the Electromagnetic (EM) Spectrum
The colors that humans perceive in an object are determined by the nature of the light reflected from the object.
A body that reflects light that is balanced in all visible wavelengths appears white to the observer. However, a body that favors reflectance in a limited range of the visible spectrum exhibits some shades of color.
For example, green objects reflect light with wavelengths primarily in the 500 to 570 nm range while absorbing most of the energy at other wavelengths.
Light that is void of color is called achromatic or monochromatic light. The only attribute of such light is its intensity, or amount.
The term gray level refers to a scalar measure of intensity that ranges from black, to grays, and finally to white.
School of Computer Science
Color Fundamentals
The human eye contains two different sorts of receptor cells: rods, which provide night-vision and cannot distinguish color, and cones, which are highly sensitive to color and in turn come in three different sorts, which respond to different wavelengths of light.
The fact that our perception of color derives from the eye’s response to three different groups of wavelengths leads to the tristimulus theory – that any color can be specified by just three values, giving the weights of each of three components. We call red, green and blue the additive primary colors.
School of Computer Science
Color Space
A color space (also called color model or color system) is a specification of a coordinate system and a subspace within that system where each color is represented by a single point.
Most color spaces in use today are oriented either toward hardware (e.g. for color monitors and printers) or toward applications where color manipulation is a goal (e.g. in the creation of color graphics for animation).
School of Computer Science
Color Space
RGB(red,green,blue)space
The RGB color space is the most important means of representing colors used in images for multimedia, because it corresponds to the way in which color is produced on computer color monitors, and it is also how color is detected by scanners.
HSV (hue, saturation, value) space
It corresponds closely with the way humans describe and interpret color
CIE L*a*b* / CIE L*u*v* space
CMYK(cyan,magenta,yellow,black)space Better for color printing
School of Computer Science
The RGB color gamut
In practice, the majority of colors perceived in the world do fall within the RGB gamut, so the RGB model provides a useful, simple and efficient way of representing colors.
A color can be represented by three values. We can write this representation in the form (r,g,b), where r, g and b are the amounts of red, green and blue light making up the color. By “amount”, we mean the proportion of pure (saturated) light of that primary.
(100%, 0%, 0%) – pure saturated primary red; (50%, 0%, 0%) – a darker red;
(100%, 50%, 100%) – mauve;
(0%, 0%, 0%) – black;
(100%, 100%, 100%) – white.
R. Gonzalez, R. Woods, “Digital Image Processing”, , 2002.
School of Computer Science
The RGB Color Space
Each color appears in its primary spectral components of red, green, and blue.
This space is based on a Cartesian Coordinate System.
The color subspace of interest is the cube, in which RGB values are at three corners; cyan, magenta, and yellow are at three other corners; black is at the origin; and white is at the corner farthest from the origin.
RGB 24-bit color cube
The different colors in this space are points on or inside the cube, and are defined by vectors extending from the origin.
In this space, the gray scale (points of equal RGB values) extends from black to white along the line joining these two points.
School of Computer Science
The RGB Color Space
256 is a very convenient number to use in a digital representation, since a single 8-bit byte can hold exactly that many different values, usually considered as numbers in the range 0 to 255. Thus, an RGB color can be represented in three bytes, or 24 bits.
The number of bits used to hold a color value is often referred to as the color depth.
The common color depths are sometimes distinguished by the terms millions of colors (24 bit), thousands of colors (16 bit) and 256 colors (8 bit).
School of Computer Science
Color Selection
School of Computer Science
The HSV Color Space
A particularly useful alternative method for representing (and manipulating) the colors of an image is known as the HSV color space. “HSV” refers to the hue, saturation and value of a pixel*.
In many ways, HSV space is a much more intuitive method of dealing with color, since it uses terms that match more closely with the way a layperson talks about color. When speaking of color conversationally, instead of characterizing a color as having 85% red, 0% green, and 90% blue, we would tend to say that the color is a “saturated magenta”. The HSV model follows this thinking process, while still giving the user precise definition and control.
—————————-
* Variations on the HSV model include HSI and HSB, in which the third component is either Intensity or Brightness, respectively.
School of Computer Science
School of Computer Science
The HSV Color Model
The hue of a pixel refers to its basic color – such as red or yellow or violet or magenta. It is usually represented in the range of 0 to 360, referring to the color’s location ( in degree ) around a circular color palette. For example, the color located at 90° corresponds to a yellow green, and pure blue is located at exactly 240° .
School of Computer Science
The HSV Color Model
Saturation is the brilliance or purity of the specific hue that is present in the pixel. If we look again at the HSV color wheel, colors on the perimeter are fully saturated, and the saturation decreases as you move to the center of the wheel.
School of Computer Science
The HSV Color Model
Value can just be thought of as the brightness of the color, although strictly speaking it is defined to be the maximum of red, green, or blue values. Trying to represent this third component means that we need to move beyond a 2D graph. The value is graphed along the third axis, with the lowest value, black, being located at the bottom of the cylinder. White, the highest brightness value, is consequently located at the opposite end.
School of Computer Science
var_R = ( R / 255 ) //RGB values = From 0 to 255 var_G = ( G / 255 )
var_B = ( B / 255 )
var_Min = min( var_R, var_G, var_B ) //Min. value of RGB var_Max = max( var_R, var_G, var_B ) //Max. value of RGB del_Max = var_Max – var_Min //Delta RGB value
V = var_Max
if ( del_Max == 0 ) {
S = del_Max / var_Max
//This is a gray, no chroma… //HSV results = From 0 to 1
//Chromatic data…
del_R = ( ( ( var_Max – var_R ) / 6 ) + ( del_Max / 2 ) ) / del_Max del_G = ( ( ( var_Max – var_G ) / 6 ) + ( del_Max / 2 ) ) / del_Max del_B = ( ( ( var_Max – var_B ) / 6 ) + ( del_Max / 2 ) ) / del_Max
if ( var_R == var_Max ) H = del_B – del_G
else if ( var_G == var_Max ) H = ( 1 / 3 ) + del_R – del_B else if ( var_B == var_Max ) H = ( 2 / 3 ) + del_G – del_R
if ( H < 0 ) ; H += 1
if ( H > 1 ) ; H -= 1 }
School of Computer Science
if ( S == 0 ) {
R = V * 255 G = V * 255 B = V * 255
var_h = H * 6
var_i = int( var_h )
var_1 = V * ( 1 – S )
var_2 = V * ( 1 – S * ( var_h – var_i ) ) var_3 = V * ( 1 – S * ( 1 – ( var_h – var_i ) ) )
if ( var_i == 0 ) { var_r = V ; var_g = var_3 ; var_b = var_1 } elseif(var_i==1){var_r=var_2;var_g=V ;var_b=var_1} elseif(var_i==2){var_r=var_1;var_g=V ;var_b=var_3} elseif(var_i==3){var_r=var_1;var_g=var_2;var_b=V } elseif(var_i==4){var_r=var_3;var_g=var_1;var_b=V }
R = var_r * 255 G = var_g * 255 B = var_b * 255 }
{var_r=V ;var_g=var_1;var_b=var_2} //RGB results = From 0 to 255
//HSV values = From 0 to 1 //RGB results = From 0 to 255
//Or … var_i = floor( var_h )
School of Computer Science
Color Interpolation Using RGB Colors
Fractional Time
HSV Colour
RGB Colour
1.0, 0.0, 0.0
0.75, 0.0, 0.25
0.5, 0.0, 0.50
0.25, 0.0, 0.75
0.0, 0.0, 1.00
Description
Bright red
Dark red-blue
Medium gray
Dark blue-red
Bright blue
School of Computer Science
Color Interpolation Using HSV Colors
Fractional Time
HSV Colour
1.0, 0.0, 0.0
0.92, 0.0, 0.0
0.83, 0.0, 0.0
0.75, 0.0, 0.0
0.67, 0.0, 0.0
RGB Colour
1.0, 0.0, 0.0
1.0, 0.0, 0.5
1.0, 0.0, 1.0
0.5, 0.0, 1.0
0.0, 0.0, 1.0
Description
Bright red
Bright red-blue
Bright purple
Bright blue-red
Bright blue
School of Computer Science
Comparison of Color Spaces
RGB is most widely used in image storage and displaying.
RGB is not perceptually uniform.
CIE consists of luminance component and two chromatic components which are more perceptually attractive.
HSV is perceptually uniform.
HSV is widely used in computer graphics and is often selected due
to its invariant properties of illumination and camera direction.
There is no agreement on which is the best choice.
School of Computer Science
Color Histogram
The color histogram serves as an effective representation of the color content of an image if the color pattern is unique compared with the rest of the data set.
The color histogram is easy to compute and effective in characterizing both the global and local distributions of colors in an image.
It is also robust to translation and rotation about the viewing axis and changes only slowly with the scale and viewing angle.
Since any pixel in the image can be described by three components in a certain color space, a histogram, i.e., the distribution of the number of pixels for each quantized bin, can be defined for each color component.
School of Computer Science
Color Histogram
Fi ( I ) The number of pixels with color i The number of pixels within image I
School of Computer Science
Color Histogram
The more bins a color histogram contains, the more discrimination power it has. However, a histogram with a large number of bins will not only increase the computational cost, but will also be inappropriate for building efficient indexes for images databases.
Furthermore, a very fine bin quantization does not necessarily improve the retrieval performance in many applications.
Need to reduce the number of bins
School of Computer Science
Color Look-up Table
Lawn Green
Pale Green
Dark Green
Light Cyan
Spring Green
Light Magenta
Marine Blue
Light Yellow
Olive Drab
School of Computer Science
Color Histogram
Reducing the number of bins –
Down-sampling the color depth / Quantization of the color space;
Use the bins that have the largest pixel numbers (since a small number of histogram bins capture the majority of pixels of an image). Such a reduction does not degrade the performance of histogram matching, but may even enhance it since small histogram bins are likely to be noisy;
Clustering methods – determine the K best colors and then calculate the number of pixels that fall in each of the K best colors.
Semantic description such as color space of X11 systems.
School of Computer Science
Histogram Intersection
Histogram Intersection is employed to measure the similarity between two histograms.
Fi ( I ) The number of pixels with color i The number of pixels within image I
N min(F(I ),F(I )) iQiD
S(IQ,ID) i1 N F(I )
School of Computer Science
Histogram Intersection
color frequency
Colors that are not present in the query image do not contribute to the intersection distance.
School of Computer Science
Color Moments
Color moments have been proved to be efficient and effective in representing color distributions of images, and have been successfully used in many retrieval systems (like IBM’s QBIC), especially when the image contains only objects.
The first order moment (mean)
The second order moment (variance)
The third order moment (skewness)
where fij is the value of the i-th color component of the image pixel j, and N is the number of pixels in the image
i Nfij j1
1 N 1 i (N (fij i)2)2
1 N 1 si (N (fij i)3)3
School of Computer Science
Color Moments
Since only 9 (three moments for each of the three color components) numbers are used to represent the color content of each image, color moments are very compact representations compared to other color features.
Due to this compactness, they may also lower the discrimination power. Usually, color moments can be used as the first pass to narrow down the search space before other sophisticated color features are used for retrieval.
School of Computer Science
Color Coherence Vector
Motivation
Color histogram does not present spatial information.
Color histogram is sensitive to both compression artifacts and camera autogain.
A special color descriptor taking spatial information into account should be proposed.
Color coherence vector (CCV) can be used to distinguish images whose color histograms are indistinguishable.
School of Computer Science
Color Coherence Vector
Two images with similar color histogram, even Their red colors! their appearances are significantly different..
G. Pass, R. Zabih, J. Miller School of Computer Science Cornell University
Color Coherence Vector
A color’s coherence is defined as the degree to which pixels of that color are members of large similarly-colored regions.
These significant regions are referred as coherent regions which are observed to be of significant importance in characterizing images.
Coherence measure classifies pi
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com