CS4185 Multimedia Technologies and Applications
Course Project
Objectives
The objectives of this course project are for students to have some hands-on experiences of multimedia programming and to develop an image retrieval application. This course project is interesting because we learn how to find a particular object (football in our case) from a set of images. You are given an image retrieval program written using C++/OpenCV, and are asked to extend it to provide additional features. This project involves first extracting different features from the input image, and then improving the image/object matching performances through different ways of combining the extracted features.
Requirements of the Course Project
This course project can be carried out as an individual or group project. The maximum number of members in each group is 3. However, we expect more work and better results from a group with more people, and the responsibility of each group member should be clearly indicated in the report.
You are given an OpenCV-based demo program. The package includes two image datasets (dataset1 and dataset2). dataset1 contains a lot of images, some of which contain footballs in them, for the Image Retrieval task. dataset2 contains only football images for the Object Detection task.
Using the given matching methods in the demo program, you can only correctly retrieve a few matched images that contain the desired object (i.e., football in this case) or locate some of the desired objects. You are asked to improve the retrieval performance of this program by adding more feature extractors.
There are two levels of requirements for the project, basic and advanced, to cater for students of different backgrounds and interests. The basic requirements are designed for all the students to practice some multimedia programming skills. The advanced requirements are for those students who would like to go further to create an application, and are more flexible in terms of what you would like to do. The basic requirements and advanced requirements account for 80% and 25%, respectively, of the grade for this project. The total final mark will be bounded by 100%.
2.1 Basic Requirements (80%)
Students are required to finish the following two tasks in the basic requirements:
Task 1: Image Retrieval
There are ten images in dataset1 that contain footballs. The filenames are: 990.jpg, 991.jpg, …, 999.jpg. In each retrieval, you pick one of these ten football images (i.e., images 990.jpg, 991.jpg, …, 999.jpg) as the input image to the program. (Note that as you pick an image for retrieval, you should leave a copy of this image in dataset1.) The program will return n images that it suspects to contain footballs, where n is set to 10 by default although you may change it. As there are a total of 10 football images in dataset1, the final retrieval performance is computed as the average of the 10 retrieval results.
· Improvement on the Precision (20%)
The target of this requirement is to achieve an average of 60% retrieval precision. This means that given an input football image, the program will return some matched images from dataset1. Among these returned images, at least 60% of them contain footballs. (30% precision gets 5% of marks, 60% precision gets 20% of marks, etc.)
· Improvement on the Recall (20%)
The target of this requirement is to be able to retrieve an average of 60% of all the images in dataset1 containing footballs. (30% recall gets 5% of marks, 60% recall gets 20% of marks, etc.)
Task 2: Object Detection
Detect and locate the football in each image in dataset2. Use the given football image (filename: football.png) as input and generate bounding boxes to indicate the locations of the football in the images, as shown in the demo program.
· Improvement on Top 10 Detection Accuracy (20%)
Top 10 accuracy refers to the situation that one of the top 10 detected bounding boxes should be a correct match with the predefined bounding box based on the intersection over union (IoU) metric. IoU is the intersected region of two bounding boxes divided by the union of the two bounding boxes. (For these two bounding boxes, one is the predefined bounding box provided by us and the other is detected by your program.) Here, for each retrieved image, if the best IoU among the top 10 returned bounding boxes is more than 0.1, we consider this image as a correct detection. The final accuracy is defined by how many retrieved images that are considered as correct detection. To be exact, if all 10 images are considered as correct detection, your algorithm has 100% accuracy. See:
http://www.mathworks.com/help/vision/ref/bboxoverlapratio.html for more information.
Note: You should use the same setting to test all the images in dataset2 and report your accuracy. (Evaluation code has already been included in the demo program. 40% accuracy gets 5% of marks, 70% accuracy gets 20% of the marks, etc.)
· Improvement on IoU (20%)
This is to try and improve the localization accuracy measured by IoU. The higher the IoU that you get, the higher the mark that you will receive. The final IoU score is computed by averaging the IoU obtained from each of the images in dataset2. 20% of IoU gets 5% of marks, 30% of IoU gets 10% of marks, 40% or above of IoU gets 20% marks.
2.2 Advanced Requirements (25%)
Students are expected to extend the program into an application. The extension may be in two directions, technical improvement and UI design. The technical improvement may include new retrieval algorithms (80+% of precision and recall gets 15% of marks), high dimensional data indexing (efficiently storing and managing the features extracted from the database, modifying the program so that it does not need to compute the features every time), retrieval algorithms for particular types of images (e.g., sunset images, images containing human faces), a crawler to obtain images from the internet, or adding semantic information to help improve the retrieval performance. A UI may include real-time display of the regions of each image being compared and their scores, or allowing users to select different objects to be retrieved from the database. Note that these are only suggested possible tasks that you may do.
Grading
The course work component contributes 40% of the final course mark/grade. Within this 40%, I will choose the best combination from one of the following for each of you:
· 15% for assignment, 25% for quiz
· 20% for assignment, 20% for quiz
· 25% for assignment, 15% for quiz
Note that we will use a PC with the following configurations to grade the course projects:
· Windows with Visual Studio 2019
· OpenCV 2.4.13
I understand that SCM students mainly use Macs. Unfortunately, we find that there are compatibility problems when installing our demo program on Macs. Hence, to address the difficulty, I have asked cslab to install the above tools in all the PCs in room MMW2410 in the cslab. So, you may use those PCs for your course project, if you like. Alternatively, you can also install Windows on your Macs. This is probably the best way to solve the problem.
Submission Details
There are two submission deadlines:
1. October 25, 2021 – basic requirements
2. November 15, 2021 – advanced requirement
By each submission deadline, each group needs to submit the following sub-directories as a zip file on Canvas. If your zip file is too large, you may submit a link where we can download your zip file.
/Program:
(1) A source sub-directory containing all the source files and the necessary files.
(2) A binary subdirectory containing the executable file of the program and relevant files, including image files or libraries. The executable file should output the retrieved results (e.g. the list of the retrieved images), precision and recall rates. Note that it is important to make sure that we may just click on the executable file to run the program. So, you may need to try the executable file on a different machine before you submit the work.
(3) A readme file with instructions on how to compile and execute the program.
/Demo:
A demo video that guides the marker through the main contributions of the work.
/Report:
The purpose of this report is to describe the main contributions of the work. In the first submission deadline, this report should describe your work on the basic requirements only. In the second submission deadline, this report should be extended to cover the advanced requirements also. Note that we will not be marking on the report itself. Instead, the report should show us what you have done so that we may grade the work appropriately. Hence, there is no need to submit a large report. It can just be a few pages providing the following information:
(1) A cover that indicates your name(s) and student ID(s)
(2) A brief description of the revised program, including the main modules and the relationship of these modules. (The description may be in the form of short paragraphs or a flow diagram.)
(3) A list of features added to the original program, including the names of the modified modules (in reference to point (2) above), brief explanations, and screen captures of the results.
(4) Responsibilities of each group member (if applicable), including
· The programmer of each added function
· The author of each major section of the report
· The person who has done the survey, group coordination, etc.
Note that we need all of the above information in order to grade the course projects fairly. So, your submission must contain the above three items. Marks will be deducted if any is missing.
2
footballs
containing
dataset
in the
images
of
number
football)
containing
AND
images
(retrieved
of
number
Recall
=
images
retrieved
of
number
football)
containing
AND
images
(retrieved
of
number
Precision
=