Deep Learning (COSC 2779)
Assignment 1: Introduction to Deep Convolutional Neural Networks
Due Date Week 7, Wednesday 8th September 2021, 05:00pm Marks 30%
1 Overview
In this assignment you will explore a real dataset to practice the typical deep learning process. The assignment is designed to help you become more confident in applying deep learning approaches. In this assignment you will:
• Develop a deep learning system to solve a real-world problem.
• Analyse the output of the algorithm(s).
• Research how to extend the DL techniques that are taught in class.
• Provide an ultimate judgement of the final trained model that you would use in a real-world setting.
To complete this assignment, you will require skills and knowledge from lecture and lab material for Weeks 1 to 6 (inclusive). You may find that you will be unable to complete some of the activities until you have completed the relevant lab work. However, you will be able to commence work on some sections. Thus, do the work you can initially, and continue to build in new features as you learn the relevant skills. A deep learning model cannot be developed within a day or two. Therefore, start early.
2 Learning Outcomes
This assessment relates to all of the learning outcomes of the course which are:
• Discuss and critically analyse a variety of neural network architectures; Evaluate and Compare approaches and algorithms on the basis of the nature of the prob- lem/task being addressed.
• Synthesise suitable solutions to address particular machine learning problems based on analysis of the problem and characteristics of the data involved.
Assessment Type
Individual assignment. Submit online via Canvas → Assign- ments → Assignment 1. Marks awarded for meeting require- ments as closely as possible. Clarifications/updates may be made via announcements/relevant discussion forums.
3
•
•
Communicate effectively with a variety of audiences through a range of modes and media, in particular to: interpret abstract theoretical propositions, choose methodologies, justify conclusions and defend professional decisions to both IT and non-IT personnel via technical reports of professional standard and technical presentations.
Develop skills for further self-directed learning in the general context of neural net- works and machine learning; Research, Discuss, and Use new and novel algorithms for solving problems; Adapt experience and knowledge to and from other computer sciences contexts such as artificial intelligence, machine learning, and software de- sign.
Assessment details
3.1 Task
Human action recognition (HAR) has a wide range of applications, such as intelligent video surveillance and environmental home monitoring, video storage and retrieval, intel- ligent human–machine interfaces, and identity recognition. Usually, HAR is performed with videos. However, processing videos on low compute devices such as IoT edge devices can be challenging and therefore it is important to develop HAR systems that can work with still images.
In this assignment, you will develop a deep convolutional neural network (CNN) to identify the actions of a person from still images. The actions we are interested in is categorized in to 21 types belonging to 5 classes.
The machine learning task we are interested in is to “Predict the action and the action-class performed by a person in an input RGB image”. Note that we are interested in predicting action for people that might not be included in the training set.
Please read the following requirements and guidelines carefully.
• You need to design a network(s) that takes in an RGB image as input, and produce two outputs: action & action-class. For higher grades (DI/HD) you should develop a single network that is capable of outputting both action & action-class.
• It is not acceptable to develop a single network that output action and then do post-processing to derive the action-class. Networks(s) should learn to output both action & action-class.
• You may use pre-trained networks as part of your solution. However, there need to be “clearly identifiable” network segment(s) that is designed and trained by you. You should show how this segment is developed (tuned) in you code.
• For higher grades (DI/HD) you should write your own data loader.
• Each element of your system should be justified using data analysis, performance
analysis and/or knowledge from relevant literature.
• You should clearly explain your evaluation framework, including how you selected appropriate performance measures, and how you determined the data splits.
2
4
•
•
Finally you need to analyse the results from your model using appropriate tech- niques and establish how adequate your model is to perform the task in real world and discuss limitation if there are any (ultimate judgement).
Predict the result for the test set.
Suggested Schedule
We expect that you will start the assignment immediately and follow a schedule similar to the one shown below. Do not fall behind, A deep learning model cannot be developed within a day or two.
• Week4: Read the specification and familiarize with the problem. Explore the data set.
• Week5: Design the experiments. Develop the data loading mechanism. Search relevant literature and read. Start writing the report.
• Week6: Develop the model – design and train. Do model analysis. Update the report.
• Week7: Do model analysis. Complete the report. Submission! 4.1 Dataset
The data set for this assignment is available on Canvas. There are the following files:
• “README.md”: Description of dataset.
• “Images.zip”: Contain all the images (test and train set).
• “S40AR train data.csv”: This data is to be used in developing the models. Use this for your own exploration and evaluation of which approach you think is “best” for this prediction task.
• “S40AR test data.csv”: You need to predict the action & action-class for this data and submit the prediction via canvas. The teaching team will use this data to evaluate the performance of the model you have developed.
• “s1234567 predictions.csv”: Shows the expected format for your predictions on the unseen test data. You should organize your predictions in this format. Any deviation from this format will result on zero marks for the results part (this include the order of images). Change the number to your student ID. Note: Your predictions should be in text format and match category labels in train data file – integer predictions will not be accepted.
The original data is from Stanford 40 Actions Dataset published with B. Yao, X. Jiang, A. Khosla, A.L. Lin, L.J. Guibas, and L. Fei-Fei. Human Action Recognition by Learning Bases of Action Attributes and Parts. Internation Conference on Computer Vision (ICCV), Barcelona, Spain. November 6-13, 2011.
3
Licence agreement: The dataset can only be used for the purpose of this assignment. Sharing or distributing this data or using this data for any other commercial or non-commercial purposes is prohibited (including uploading the data to a public GitHub repository).
5 Submission
You have to submit all the relevant material as listed below via Canvas.
1. A report (of no more than 3 pages ) critically analysing your approach and ultimate judgement. Should be in PDF format.
2. A set of predictions from your ultimate judgement. Should be in the format and order shown in s1234567 predictions.csv. This file will be auto marked and any deviation from the given format will result in zero marks.
3. Your code (Jupyter notebooks) used to perform your analysis. Should be a ZIP file containing all the support files.
The submission portal on canvas consists of two sub-pages. page one for report submission and the second page for code and other file submission. More information is provided on canvas. Include only source code in a zip file containing your name. We strongly recommend you to attach a README file with instructions on how to run your application. Make sure that your assignment can run only with the code included in your zip file! Include a PDF version of your report.
After the due date, you will have 5 business days to submit your assignment as a late submission. Late submissions will incur a penalty of 10% per day. After these five days, Canvas will be closed and you will lose ALL the assignment marks.
Assessment declaration:
When you submit work electronically, you agree to the assessment declaration – https:// www.rmit.edu.au/students/student-essentials/assessment-and-exams/assessment/ assessment-declaration
6 Teams
Not relevant. This is an individual assignment.
7 Academic integrity and plagiarism (standard warning)
Academic integrity is about honest presentation of your academic work. It means ac- knowledging the work of others while developing your own insights, knowledge and ideas. You should take extreme care that you have:
• Acknowledged words, data, diagrams, models, frameworks and/or ideas of others you have quoted (i.e. directly copied), summarised, paraphrased, discussed or men- tioned in your assessment through the appropriate referencing methods
4
• Provided a reference list of the publication details so your reader can locate the source if necessary. This includes material taken from Internet sites. If you do not acknowledge the sources of your material, you may be accused of plagiarism because you have passed off the work and ideas of another person without appropriate referencing, as if they were your own.
RMIT University treats plagiarism as a very serious offence constituting misconduct. Plagiarism covers a variety of inappropriate behaviours, including:
• Failure to properly document a source
• Copyright material from the internet or databases • Collusion between students
For further information on our policies and procedures, please refer to the following:
https://www.rmit.edu.au/students/student-essentials/rights-and-responsibilities/ academic-integrity.
8 Marking guidelines
A detailed rubric is attached on canvas. In summary: • Approach 50%;
• Ultimate Judgment & Analysis 20%;
• Performance on test set (Unseen data) 20%;
• Implementation & Report Presentation 10%;
Approach: You are required to use a suitable approach to find a predictive model. You may use any form of ML paradigm, including: Regression or Classification. Each element of the approach need to be justified using data analysis, performance analysis and/or published work in literature. This assignment isn’t just about your code or model, but the thought process behind your work. The elements of your approach may include:
• Setting up the evaluation framework
• Selecting CNN architecture, loss function and optimization procedure. • Hyper-parameter setting and tuning
• Identify problem specific issues/properties and solutions
Ultimate Judgement: You must make an ultimate judgement of the “best” model that you would use and recommend in a real-world setting for this problem. It is up to you to determine the criteria by which you evaluate your model and determine what is means to be “the best model”. You need to provide evidence to support your ultimate judgement and discuss limitation of your approach/ultimate model if there are any.
Performance on test set (Unseen data): You must use the model chosen in your ulti- mate judgement to predict the action on unseen testing data (provided in test data.csv).
5
Your ultimate prediction will be evaluated, and the performance of all of the ultimate judgements will be published.
Critical Analysis & Report
Finally, you must compile a report describing and analysing the approach that you have taken to find a suitable model and make your ultimate judgement. Your report must be no longer that 3 pages . Any over length content, or content outside of these requirements will not be marked. For example, if you report is too long, ONLY the first 3 pages pages of text will be read and marked.
A report template is provided to you on canvas. It is recommended to use this format. In general your report should be in 10pt times new roman font with 1.15 line-spacing.
In this report you should describe elements such as:
• Your final selected approach
• Why you selected this approach
• Parameter settings and other approaches you have tried.
• Limitation and improvements that are required for real-world implantation.
This will allow us to understand your rationale. We encourage you to explore this problem and not just focus on maximising a single performance metric. By the end of your report, we should be convinced that of your ultimate judgement and that you have considered all reasonable aspects in investigating this problem.
Remember that good analysis provides factual statements, evidence and justifications for conclusions that you draw. A statements such as:
“I did xyz because I felt that it was good”
is not analysis. This is an unjustified opinion. Instead, you should aim for statements such as:
“I did xyz because it is more efficient. It is more efficient because …”
6