MULTIMEDIA RETRIEVAL
Semester 1, 2022
Sketch based Image Retrieval (SBIR)
Introduction to Query by Sketch
Copyright By PowCoder代写 加微信 powcoder
Conventional Methods Edge representation Large scale SBIR
Deep Learning Methods Categorical SBIR Fine-grained SBIR
School of Computer Science
Query Types for Image Retrieval
Query by words: keywords/phrases/sentences
Query by Example
Query by Sketch: color sketch and line sketch
School of Computer Science
Query by Sketch
Draw a Sketch, Then Search!
With slides from .
Y. Cao, et al., MindFinder: Interactive Sketch-based Image Search on Millions of Images, ACM MM 2010. C. Xiao, et al., IdeaPanel: A Large Scale Interactive Sketch-based ImageSearch System, ICMR 2015.
School of Computer Science
Draw a Sketch, Then Search!
School of Computer Science
School of Computer Science
School of Computer Science
UI of MindFinder
School of Computer Science
MindFinder Demo
School of Computer Science
Challenges
How to bridge the gap between a natural image and a query sketch
School of Computer Science
Challenges
How to build an efficient index to support real-time response in a million or billion level database
School of Computer Science
Sketch Representation
Challenges
Extracting representative curves/edges Shape content
Shape position
Matching efficiency and effectiveness Scalability: suitable indexing mechanism
School of Computer Science
Feature lines
Edge segment based features
Edge histogram
The distribution of gradient
orientations in a cell
Tensor descriptor
a single vector in a cell that is as
parallel as possible to the image gradients in a cell Edgel for edge indexing
Hash algorithm
M. Eitz, et al., A descriptor for large scale image retrieval based on sketched feature lines, Eurographics Symposium on Sketch-Based Interfaces and Modeling (SBIM), 2009. Y. Cao, et al., Edgel index for large-scale sketch-based image search, CVPR 2011.
School of Computer Science
Edge-segment based SBIR
Edge Segment Extraction
Y. Chans, et al., A feature-based approach for image retrieval by sketch. SPIE Symposium on Multimedia Storage and Archiving Systems II
School of Computer Science
Edge-segment based SBIR
Y. Chans, et al., A feature-based approach for image retrieval by sketch. SPIE Symposium on Multimedia Storage and Archiving Systems II
School of Computer Science
Edge-segment based SBIR
Hough transform is to detect the simple shapes such as lines and circles in an image.
cv2.HoughLinesP()
School of Computer Science
Edge-segment based SBIR
Identify line segments longer than a length threshold
School of Computer Science
Edge-segment based SBIR
Similarity measurement
School of Computer Science
Edge-segment based SBIR
Similarity measurement
Similarity:
School of Computer Science
Edge-segment based SBIR
Experimental Dataset
137 color photo and computer generated images
School of Computer Science
Edge-segment based SBIR
Experimental Results
School of Computer Science
Edge-segment based SBIR
Experimental Results
Sketches from 5 subjects for 16 images
School of Computer Science
Edge-segment based SBIR
Experimental Results
School of Computer Science
Towards a million images
Sketch representation
Sketch/stroke segments
The direction of a stroke relative to its position
Edge histogram descriptor Tensor descriptor
Cell Ci, j
M. Eitz, et al., A descriptor for large scale image retrieval based on sketched feature lines, Eurographics Symposium on Sketch-Based Interfaces and Modeling (SBIM), 2009.
School of Computer Science
Towards a million images
Edge histogram descriptor (EHD) Image I
d-dimensional gradient histogram of Ci,j
Cell Ci, j
Distance between gradient histograms:
guv is the gradient of Image I
School of Computer Science
EHD in MPEG-7
Bin[i, dominant] ++ Bin[i] = {b1, b2, b3, b4, b5}
Bin[i].normalisation EHD= [Bin[1], …, Bin[16]]
B. Ahmed, et al.,Image Retrieval based on Edge Histogram Descriptor of MPEG-7, Eurasian Journal of Science & Engineering, 2020.
School of Computer Science
Towards a million images
Tensor descriptor
Aims to characterize the structure in a cell
Main direction
Matrix Gij contains the sum of outer products of gradients in cell Cij and is commonly referred to as the structure tensor
School of Computer Science
Towards a million images
Tensor descriptor
Tensor is normalized with Frobenius norm
School of Computer Science
Towards a million images
Experimental Results
a query in the 1.5 million image database (from Flickr) takes between 0.4 and 3.5 seconds depending on the sparsity of the user sketch
School of Computer Science
Towards a million images
School of Computer Science
Towards millions of images: Edgel
Each image consists of a set of edge pixel (edge segments), edgel
An Edgel Its position
Its gradient orientation
The basic Chamfer Distance between a
database image and the query sketch is
The number of edgels in image 𝒟
Y. Cao, et al., Edgel index for large-scale sketch-based image search, CVPR 2011.
School of Computer Science
Towards millions of images: Edgel
Effective matching and indexing
Edgel: Edge Pixel triplet: p=(x, y, θ) as a “word”
The orientation space is equally quantified into 6 bins/channels: −150 ∼ 150, 150 ∼ 450, . . . , 1350 ∼ 1650
(6, 7, 450) –> (6, 7, 1)
Y. Cao, et al., Edgel index for large-scale sketch-based image search, CVPR 2011.
School of Computer Science
Edgel Index for SBIR
Chamfer Matching Query sketch Q
Database image D
(6, 7, 450) –> (6, 7, 1)
Y. Cao, et al., Edgel index for large-scale sketch-based image search, CVPR 2011.
School of Computer Science
Edgel Index for SBIR
Oriented Chamfer Matching Query sketch Q
Database image D
(6, 7, 450) –> (6, 7, 1)
Y. Cao, et al., Edgel index for large-scale sketch-based image search, CVPR 2011.
School of Computer Science
Edgel Index for SBIR
Y. Cao, et al., Edgel index for large-scale sketch-based image search, CVPR 2011.
School of Computer Science
Edgel Index for SBIR
Similarity with Hit maps
Y. Cao, et al., Edgel index for large-scale sketch-based image search, CVPR 2011.
School of Computer Science
Index Structure
Index of Database -> Query:
Index of Database -> Query:
Sketch Query :
distance map
=> expanded sketch query
Inverted List
Edgel Dictionary:
Salient Curve Identification
Image Canny
Boundary Saliency
Edgel ‐> (x, y, Θ) Vocabulary: 500 x 500 x 6 orientation = 1,500,000
Vocabulary: 200 x 200 x 6 orientation = 240K codes
System Scalability
Database Size
Memory Cost
Response Time
2.1 Million
10 Million
School of Computer Science
Indexing Billions of Images
X. Sun, et al., Indexing billions of images for sketch-based retrieval. ACM MM 2013.
School of Computer Science
Indexing Billions of Images
More compact representation and efficient matching
X. Sun, et al., Indexing billions of images for sketch-based retrieval. ACM MM 2013.
School of Computer Science
Deep Learning based SBIR
Category-level SBIR
Fine-grained: Instance-level SBIR
School of Computer Science
Category-level SBIR
Deep Features
https://coolgpu.github.io/coolgpu_blog/github/pages/2020/10/04/convolution.html https://medium.com/analytics-vidhya/introduction-to-convolutional-neural-networks-c50f41e3bc66
https://www.freecodecamp.org/news/an-intuitive-guide-to-convolutional-neural-networks-260c2de0a050
School of Computer Science
https://www.freecodecamp.org/news/an-intuitive-guide-to-convolutional-neural-networks-260c2de0a050
School of Computer Science
Category-level SBIR
Sketch recognition
https://www.topbots.com/deep-transfer-learning-image-classification/
School of Computer Science
Category-level SBIR
Sketch recognition
TU-Berlin sketch dataset
250 categories with 80 sketches per category
Collected on Amazon Mechanical Turk (AMT) from 1,350 participants Providing a diversity of both categories and sketching styles within each category.
Q. Yu, et al., Sketch-a-Net that Beats Humans. BMVC 2015.
School of Computer Science
Category-level SBIR
Sketch recognition
Modeling stroke order
Q. Yu, et al., Sketch-a-Net that Beats Humans. BMVC 2015.
School of Computer Science
Shoe Search with Deep Learning
Cross modal: sketches vs color pixels
Abstract
Fine-grained: category level retrieval is not enough
Requiring sufficient paired training data
Y. Z. Song, et al., Sketch me that Shoe. CVPR 2016.
School of Computer Science
Sketch Me That Shoe
Feature Extraction
Similarity based Retrieval
School of Computer Science
Sketch Me That Shoe
Encoder fθ(p+)
Encoder fθ(p-)
Triplet Loss
School of Computer Science
Experimental Dataset
Instance-level SBIR dataset
716 sketch-photo pairs of shoe and chairs
Drawn by amateurs on touch-screen devices
32,000 groundtruth triplet ranking annotation
Given a sketch, ranking which of two photos is more similar
School of Computer Science
Experimental Results
School of Computer Science
Experimental Results
School of Computer Science
StyleMeUp: Towards Style-agnostic SBIR
Existing methods did not consider drawing style of individuals
Style Diversity in Sketches User2/
User1/ Style1
A. Sain, et al., StyleMeUp: Towards Style-agnostic Sketch Based Image Retrieval. CVPR 2021.
School of Computer Science
StyleMeUp: Towards Style-agnostic SBIR
Slides from A. Sain, et al., StyleMeUp: Towards Style-agnostic Sketch Based Image Retrieval. CVPR 2021.
School of Computer Science
Experimental Dataset
Fine-grained SBIR
QMUL ShoeV2
2000 sketches + 200 photos
QMUL ChairV2
7630 sketches + 2000 photos
Categorical SBIR
Sketchy (ext.)
75k sketches across 125 categories with about 73k images
TUBerlin (ext.)
250 object categories with 80 free-hand sketches per category
School of Computer Science
Experimental Results
School of Computer Science
Experimental Results
School of Computer Science
Experimental Results
QMUL ChairV2
School of Computer Science
Experimental Results
QMUL ShoeV2
School of Computer Science
Need To Know
SBIR and its applications
Sketch features
Deep learning based image retrieval Deep learning for SBIR
School of Computer Science
References
Sketch-based image retrieval: benchmark and bag-of-features descriptors
IEEE Transactions on Visualization and Computer Graphics (TVCG), 17(11): 1624-1636, 2011.
http://cybertron.cg.tu- berlin.de/eitz/tvcg_benchmark/index.html
Sketch based deep learning
https://github.com/topics/sketch-based-image-retrieval
https://github.com/qyzdao/Sketch-Based-Deep- Learning
School of Computer Science
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com