CS代考 MM 2010. C. Xiao, et al., IdeaPanel: A Large Scale Interactive Sketch-based

MULTIMEDIA RETRIEVAL
Semester 1, 2022
Sketch based Image Retrieval (SBIR)
 Introduction to Query by Sketch

Copyright By PowCoder代写 加微信 powcoder

 Conventional Methods Edge representation Large scale SBIR
 Deep Learning Methods Categorical SBIR Fine-grained SBIR
School of Computer Science

Query Types for Image Retrieval
 Query by words: keywords/phrases/sentences
 Query by Example
 Query by Sketch: color sketch and line sketch
School of Computer Science
Query by Sketch
 Draw a Sketch, Then Search!
With slides from .
Y. Cao, et al., MindFinder: Interactive Sketch-based Image Search on Millions of Images, ACM MM 2010. C. Xiao, et al., IdeaPanel: A Large Scale Interactive Sketch-based ImageSearch System, ICMR 2015.
School of Computer Science

Draw a Sketch, Then Search!
School of Computer Science
School of Computer Science

School of Computer Science
UI of MindFinder
School of Computer Science

MindFinder Demo
School of Computer Science
Challenges
 How to bridge the gap between a natural image and a query sketch
School of Computer Science

Challenges
 How to build an efficient index to support real-time response in a million or billion level database
School of Computer Science
Sketch Representation
 Challenges
Extracting representative curves/edges  Shape content
 Shape position
Matching efficiency and effectiveness Scalability: suitable indexing mechanism
School of Computer Science

 Feature lines
Edge segment based features
Edge histogram
 The distribution of gradient
orientations in a cell
Tensor descriptor
 a single vector in a cell that is as
parallel as possible to the image gradients in a cell  Edgel for edge indexing
 Hash algorithm
M. Eitz, et al., A descriptor for large scale image retrieval based on sketched feature lines, Eurographics Symposium on Sketch-Based Interfaces and Modeling (SBIM), 2009. Y. Cao, et al., Edgel index for large-scale sketch-based image search, CVPR 2011.
School of Computer Science
Edge-segment based SBIR
 Edge Segment Extraction
Y. Chans, et al., A feature-based approach for image retrieval by sketch. SPIE Symposium on Multimedia Storage and Archiving Systems II
School of Computer Science

Edge-segment based SBIR
Y. Chans, et al., A feature-based approach for image retrieval by sketch. SPIE Symposium on Multimedia Storage and Archiving Systems II
School of Computer Science
Edge-segment based SBIR
 Hough transform is to detect the simple shapes such as lines and circles in an image.
cv2.HoughLinesP()

School of Computer Science

Edge-segment based SBIR
 Identify line segments longer than a length threshold

School of Computer Science
Edge-segment based SBIR
 Similarity measurement
School of Computer Science

Edge-segment based SBIR
 Similarity measurement
Similarity:
School of Computer Science
Edge-segment based SBIR
 Experimental Dataset
 137 color photo and computer generated images
School of Computer Science

Edge-segment based SBIR
 Experimental Results
School of Computer Science
Edge-segment based SBIR
 Experimental Results
 Sketches from 5 subjects for 16 images
School of Computer Science

Edge-segment based SBIR
 Experimental Results
School of Computer Science
Towards a million images
 Sketch representation
Sketch/stroke segments
 The direction of a stroke relative to its position
 Edge histogram descriptor  Tensor descriptor
Cell Ci, j
M. Eitz, et al., A descriptor for large scale image retrieval based on sketched feature lines, Eurographics Symposium on Sketch-Based Interfaces and Modeling (SBIM), 2009.
School of Computer Science

Towards a million images
 Edge histogram descriptor (EHD) Image I
d-dimensional gradient histogram of Ci,j
Cell Ci, j
Distance between gradient histograms:
guv is the gradient of Image I
School of Computer Science
EHD in MPEG-7
Bin[i, dominant] ++ Bin[i] = {b1, b2, b3, b4, b5}
Bin[i].normalisation EHD= [Bin[1], …, Bin[16]]
B. Ahmed, et al.,Image Retrieval based on Edge Histogram Descriptor of MPEG-7, Eurasian Journal of Science & Engineering, 2020.
School of Computer Science

Towards a million images
 Tensor descriptor
 Aims to characterize the structure in a cell
 Main direction
 Matrix Gij contains the sum of outer products of gradients in cell Cij and is commonly referred to as the structure tensor
School of Computer Science
Towards a million images
 Tensor descriptor
 Tensor is normalized with Frobenius norm
School of Computer Science

Towards a million images
 Experimental Results
 a query in the 1.5 million image database (from Flickr) takes between 0.4 and 3.5 seconds depending on the sparsity of the user sketch
School of Computer Science
Towards a million images
School of Computer Science

Towards millions of images: Edgel
 Each image consists of a set of edge pixel (edge segments), edgel
An Edgel  Its position
 Its gradient orientation
 The basic Chamfer Distance between a
database image and the query sketch is
The number of edgels in image 𝒟
Y. Cao, et al., Edgel index for large-scale sketch-based image search, CVPR 2011.
School of Computer Science
Towards millions of images: Edgel
 Effective matching and indexing
 Edgel: Edge Pixel triplet: p=(x, y, θ) as a “word”
 The orientation space is equally quantified into 6 bins/channels: −150 ∼ 150, 150 ∼ 450, . . . , 1350 ∼ 1650
(6, 7, 450) –> (6, 7, 1)
Y. Cao, et al., Edgel index for large-scale sketch-based image search, CVPR 2011.
School of Computer Science

Edgel Index for SBIR
 Chamfer Matching Query sketch Q
Database image D
(6, 7, 450) –> (6, 7, 1)
Y. Cao, et al., Edgel index for large-scale sketch-based image search, CVPR 2011.
School of Computer Science
Edgel Index for SBIR
 Oriented Chamfer Matching Query sketch Q
Database image D
(6, 7, 450) –> (6, 7, 1)
Y. Cao, et al., Edgel index for large-scale sketch-based image search, CVPR 2011.
School of Computer Science

Edgel Index for SBIR
Y. Cao, et al., Edgel index for large-scale sketch-based image search, CVPR 2011.
School of Computer Science
Edgel Index for SBIR
 Similarity with Hit maps
Y. Cao, et al., Edgel index for large-scale sketch-based image search, CVPR 2011.
School of Computer Science

Index Structure
Index of Database -> Query:
Index of Database -> Query:
Sketch Query :
distance map
=> expanded sketch query
Inverted List
Edgel Dictionary:
Salient Curve Identification
Image Canny
Boundary Saliency
Edgel ‐> (x, y, Θ) Vocabulary: 500 x 500 x 6 orientation = 1,500,000
Vocabulary: 200 x 200 x 6 orientation = 240K codes

System Scalability
Database Size
Memory Cost
Response Time
2.1 Million
10 Million
School of Computer Science
Indexing Billions of Images
X. Sun, et al., Indexing billions of images for sketch-based retrieval. ACM MM 2013.
School of Computer Science

Indexing Billions of Images
 More compact representation and efficient matching
X. Sun, et al., Indexing billions of images for sketch-based retrieval. ACM MM 2013.
School of Computer Science
Deep Learning based SBIR
 Category-level SBIR
 Fine-grained: Instance-level SBIR
School of Computer Science

Category-level SBIR
Deep Features
https://coolgpu.github.io/coolgpu_blog/github/pages/2020/10/04/convolution.html https://medium.com/analytics-vidhya/introduction-to-convolutional-neural-networks-c50f41e3bc66
https://www.freecodecamp.org/news/an-intuitive-guide-to-convolutional-neural-networks-260c2de0a050
School of Computer Science
https://www.freecodecamp.org/news/an-intuitive-guide-to-convolutional-neural-networks-260c2de0a050
School of Computer Science

Category-level SBIR
 Sketch recognition
https://www.topbots.com/deep-transfer-learning-image-classification/
School of Computer Science
Category-level SBIR
 Sketch recognition
 TU-Berlin sketch dataset
 250 categories with 80 sketches per category
 Collected on Amazon Mechanical Turk (AMT) from 1,350 participants  Providing a diversity of both categories and sketching styles within each category.
Q. Yu, et al., Sketch-a-Net that Beats Humans. BMVC 2015.
School of Computer Science

Category-level SBIR
 Sketch recognition
 Modeling stroke order
Q. Yu, et al., Sketch-a-Net that Beats Humans. BMVC 2015.
School of Computer Science
Shoe Search with Deep Learning
 Cross modal: sketches vs color pixels
 Abstract
 Fine-grained: category level retrieval is not enough
 Requiring sufficient paired training data
Y. Z. Song, et al., Sketch me that Shoe. CVPR 2016.
School of Computer Science

Sketch Me That Shoe
Feature Extraction
Similarity based Retrieval
School of Computer Science
Sketch Me That Shoe
Encoder fθ(p+)
Encoder fθ(p-)
Triplet Loss
School of Computer Science

Experimental Dataset
 Instance-level SBIR dataset
 716 sketch-photo pairs of shoe and chairs
Drawn by amateurs on touch-screen devices
 32,000 groundtruth triplet ranking annotation
 Given a sketch, ranking which of two photos is more similar
School of Computer Science
Experimental Results
School of Computer Science

Experimental Results
School of Computer Science
StyleMeUp: Towards Style-agnostic SBIR
 Existing methods did not consider drawing style of individuals
Style Diversity in Sketches User2/
User1/ Style1
A. Sain, et al., StyleMeUp: Towards Style-agnostic Sketch Based Image Retrieval. CVPR 2021.
School of Computer Science

StyleMeUp: Towards Style-agnostic SBIR
Slides from A. Sain, et al., StyleMeUp: Towards Style-agnostic Sketch Based Image Retrieval. CVPR 2021.
School of Computer Science
Experimental Dataset
 Fine-grained SBIR
 QMUL ShoeV2
 2000 sketches + 200 photos
 QMUL ChairV2
 7630 sketches + 2000 photos
 Categorical SBIR
 Sketchy (ext.)
 75k sketches across 125 categories with about 73k images
 TUBerlin (ext.)
 250 object categories with 80 free-hand sketches per category
School of Computer Science

Experimental Results
School of Computer Science
Experimental Results
School of Computer Science

Experimental Results
 QMUL ChairV2
School of Computer Science
Experimental Results
 QMUL ShoeV2
School of Computer Science

Need To Know
 SBIR and its applications
 Sketch features
 Deep learning based image retrieval  Deep learning for SBIR
School of Computer Science
References
 Sketch-based image retrieval: benchmark and bag-of-features descriptors
 IEEE Transactions on Visualization and Computer Graphics (TVCG), 17(11): 1624-1636, 2011.
 http://cybertron.cg.tu- berlin.de/eitz/tvcg_benchmark/index.html
 Sketch based deep learning
 https://github.com/topics/sketch-based-image-retrieval
 https://github.com/qyzdao/Sketch-Based-Deep- Learning
School of Computer Science

程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com