1
1. 2.
3. 4.
2 2.1
Basic Requirements
Programming language: Matlab (recommended), or other languages (Python, etc.)
Implement your code under Code folder, and read data from Data folder. Under Data folder,
Database contains all the images to retrieve from.
Final project report under name report.pdf at the project root folder. Total credits for the project: 20pt
Part I: Image Retrieval (10pt) Image Feature Extraction (4pt)
Multimedia Information Systems CSE408, Spring 2019 P02: Image Retrieval and Embedding
Implement functions that takes in one image, and several hyper-parameters (based on the specific feature you are using), return a fixed length feature vector. Pick four from the following image feature extraction methods to implement.
1. Average pixel color;
2. Spatial grid of average pixel colors;
3. Color histograms;
4. Edge extraction and compute the edge energy in spatial grid;
5. Shape context: assume the center of image to be the center of the shape; 6. Histograms of Oriented Gradient (HoG).
2.2 Database feature extraction (4pt)
Apply the four feature extraction methods you implemented onto each image from the Database folder, and create your feature library.
2.3 An interface for image retrieval (2pt)
Implement an interface (graphic interface preferred, command-line is also okay) that takes in one query image and return the top 10 images from your feature library. The user should be able to indicate which image feature extraction method is used and which distance measure (SSD or Angle between vectors) is used.
1
P02: Image Retrieval and Embedding 2
2.4 Test your implementation
Test your system with the images (4.png, 25.png, 39.png, 63.png, 129.png, 147.png, 252.png, 374.png, 540.png, 572.png) from Database folder (assume they are the query images), and screen capture your top 10 closest images retrieved for each image query. Qualitatively evaluate your system¡¯s performance and write them up in your report. Especially, are their evolution family members got successfully retrieved? Why or why not?
Also, test your system with other images from online resources, could be natural images.
3 Part II: Image Embedding (6pt)
Another way to show how well a certain image feature work with the image data is to embed the images onto a lower dimensional space (2-d or 3-d) based on their feature distances. In such a manner, developer could have a straight perception of how well the feature works. In this task, we are going to use the deep neural net based image features, which is extracted from a pre-trained neural net of millions annotated image data. The idea is to mimic human vision neural system for image classification and retrieval.
3.1 PCA feature embedding
Read in the DeepNetFeature/feature vgg f.mat, which contains a matlab structure that has two ele- ments: 1) feat: a 4096 elements Deep Neural Net feature vector, and 2) name: corresponding image name from the Database folder.
Use Principle Component Analysis method (pca() in matlab) to compute the top two principle components of the data, and plot the data based on their principle components.
3.2 Test your implementation
Test your implementation and capture the plot, put it in your report. Qualitatively evaluate your plot and write them up in your report. A sample embedding looks like Fig. 1.
4 Report Requirements (4pt)
Please include the following analysis in your report.
1. Make sure to expalin where the algorithms worked and where they didn¡¯t and why. You are encouraged to use both text and plots to explain your observations.
2. Include retrieval tests output and embedding output.
3. Include qualitative analysis. You could also propose evaluation metrics and do quantitative analysis.
5 Submission Instruction
Please place your answers under one .zip file with a formatted file name: P02 ASUID.zip (for example, if your asu ID is 101010101010, then the file name should be P02 101010101010.zip), and submit on Canvas.
P02: Image Retrieval and Embedding
3
Figure 1: A sample embedding.
The .zip file shall include one folder with Matlab source code under name ¡°code¡± and one report in .pdf format.
This assignment is due on 10pm March 26th. Submission will be accepted after deadline with a 50% reduction.