Deep Learning
COSC 2779
Assignment 2
Assessment Type Individual assignment. Submit online via Canvas → Assign-
ments → Assignment 2. Marks awarded for meeting require-
ments as closely as possible. Clarifications/updates may be
made via announcements/relevant discussion forums.
Due Date Week 12, Friday 15 October 2021, 05:00pm
Marks 50%
1 Overview
In this assignment you will design and create an end-to-end deep learning system for
a real-world problem. This assignment is designed for you to apply and practice skills
of critical analysis and evaluation to circumstances similar to those found in real-world
problems. This is a Individual project.
In this assignment you will:
• Design and Create an end-to-end machine learning system.
• Apply multiple algorithms to a real-world machine learning problem.
• Analyse and Evaluate the output of the algorithms.
• Research into extending techniques that are taught in class.
• Provide an ultimate judgement of the final trained model(s) that you would use in
a real-world setting.
This assignment has the following deliverables:
1. A report (of no more than 3 pages , plus up to 2 pages for appendices) critically
analysing your approach and ultimate judgement.
2. Your Python scripts, Jupyter notebooks, and software used to build your learning
system and produce the models and results.
2 Learning Outcomes
This assessment relates to all of the learning outcomes of the course which are:
• Discuss and critically analyse a variety of neural network architectures; Evaluate
and Compare approaches and algorithms on the basis of the nature of the prob-
lem/task being addressed.
• Synthesise suitable solutions to address particular machine learning problems based
on analysis of the problem and characteristics of the data involved.
• Communicate effectively with a variety of audiences through a range of modes
and media, in particular to: interpret abstract theoretical propositions, choose
methodologies, justify conclusions and defend professional decisions to both IT
and non-IT personnel via technical reports of professional standard and technical
presentations.
• Develop skills for further self-directed learning in the general context of neural net-
works and machine learning; Research, Discuss, and Use new and novel algorithms
for solving problems; Adapt experience and knowledge to and from other computer
sciences contexts such as artificial intelligence, machine learning, and software de-
sign.
3 Assessment details
3.1 Task
Using deep learning in real-world settings involves more than just running a data set
through a particular algorithm. In this assignment, you will design, analyse and evaluate
a complete machine learning system.
The key aspect of this assignment is the design , analysis , and evaluation of your
methodology, investigation, and results. This assignment focuses on both the accuracy
of your model, and your understanding of your approach and model.
For this assignment you have a choice of your project. You may select this project
from the list in Section 4, or you may negotiate a project with the course co-ordinator.
Regardless of the problem you choose, you must conduct the following tasks:
1. Conduct a review to identify the most suitable approaches to solve the problem.
2. Investigate various Deep Learning solutions to the problem.
3. Make an ultimate judgement.
4. Evaluate your ultimate judgement against independent testing data.
5. Produce a report of your design, investigation, evaluation and findings.
4 Suggested Projects
4.1 Determining Support for Rumours
Media is full of false claims. Even Oxford Dictionaries named “post-truth” as the word
of the year. This makes it more important than ever to build systems that can identify
the truth of a story, and the kind of discourse there is around it. Here, rumour is defined
as a “circulating story of questionable veracity, which is apparently credible but hard to
verify, and produces sufficient skepticism and/or anxiety so as to motivate finding out
the actual truth” [Derczynski et. al, 2017].
While breaking news unfold, gathering opinions and evidence from as many sources
as possible as communities react becomes crucial to determine the veracity of rumours
2
and consequently reduce the impact of the spread of misinformation. Therefore, in deter-
mining rumour veracity, we can first try to figure out whether the comments made about
a given post (by other individuals on a given platform) support, deny, query or comment.
An example of this first step is given below:
• u1: These are not timid colours; soldiers back guarding Tomb of Unknown Soldier
after today’s shooting #StandforCanada PICTURE [support]
– u2: @u1 Apparently a hoax. Best to take Tweet down. [deny]
– u3: @u1 This photo was taken this morning, before the shooting. [deny]
– u4: @u1 I dont believe there are soldiers guarding this area right now. [deny]
– u5: @u4 wondered as well. Ive reached out to someone who would know just
to confirm that. Hopefully get response soon. [comment]
– u4: @u5 ok, thanks. [comment]
This project is to develop a Deep learning model to classify response tweets into one
of the following four categories:
• Support: the author of the response supports the veracity of the rumour they are
responding to.
• Deny: the author of the response denies the veracity of the rumour they are re-
sponding to.
• Query: the author of the response asks for additional evidence in relation to the
veracity of the rumour they are responding to.
• Comment: the author of the response makes their own comment without a clear
contribution to assessing the veracity of the rumour they are responding to.
The original dataset is from “Gorrell, G., Kochkina, E., Liakata, M., Aker, A., Zu-
biaga, A., Bontcheva, K. and Derczynski, L., 2019, June. SemEval-2019 task 7: Ru-
mourEval, determining rumour veracity and support for rumours. In Proceedings of the
13th International Workshop on Semantic Evaluation”
The data set is available on canvas. This data set can be combined with other data
sets that you might obtain from the internet to improve performance.
Licence agreement: The dataset can only be used for the purpose of this assignment.
Sharing or distributing this data or using this data for any other commercial or non-
commercial purposes is prohibited.
requirements
• Develop a DL based solution and demonstrate your knowledge on advanced DL
techniques when applicable.
• You may use pretrained networks in parts of your solution. However the entire
solution should not be a pretrained network. There should be clearly identifiable
sections of your approach that is trained using the provided data by yourself.
• A thorough investigation should be conducted to check the strengths and weaknesses
of your model when applied to real-world data. You should use independent test
data to conduct this investigation which may be: collected from the internet yourself
or tweets made up by you simulating real scenarios.
3
4.2 Inferring Distance in Images
Inferring Distance (or depth) from images is an important problem in computer vi-
sion with many applications including surveillance, autonomous cars, VR/AR, and tele-
presence. With the current pandemic situation such technology can also be utilized to
enforce social distancing etc.
When depth has to be estimated from camera images, stereo vision is a common
choice. These methods treat depth estimation as a purely geometrical problem, generally
completely ignoring the content of the images. On the other hand, when only a single
image is available, it is not possible to use geometry. Instead, algorithms have to rely on
pictorial cues: cues that indicate depth within a single image, such as the apparent size of
known objects. Pictorial cues require knowledge of the environment, which makes them
difficult to program. As a result, pictorial cues have seen relatively little use in computer
vision until recently when deep learning became popular.
This project is to develop a Deep learning model based system to estimate the distance
from the camera to multiple-people in the field of view. A data set collected by a RMIT
researcher is made available to you. This dataset contains images and depth ground-truth
under limited environmental scenarios with only one person per frame.
The data set is available on canvas. This data set can be combined with other data
sets that you might obtain from the internet to improve performance.
Licence agreement: The dataset can only be used for the purpose of this assignment.
Sharing or distributing this data or using this data for any other commercial or non-
commercial purposes is prohibited.
requirements
• The system should be able to predict depth for images with multiple people in the
frame. For example if there are two people on a test frame, the system should
predict the distance for both of them.
• A regression framework that can just predict depth of one person in the image will
not be considered as adequate.
• The images in the provided dataset also has information on the bounding box of
the person in frame. However this information will NOT be avaialbe at test time.
• You may use pretrained networks in parts of your solution. However the entire
solution should not be a pretrained network. There should be clearly identifiable
sections of your approach that is trained using the provided data by yourself.
• A thorough investigation should be conducted to check the strengths and weaknesses
of your model when applied to real-world data. You should use independent test
data to conduct this investigation which may be collected from the Internet yourself
or captured by you simulating real scenarios.
4
4.3 Negotiated Project
You may propose and negotiate a project and machine learning problem to investigate,
with the course co-ordinator. This project must meet a number of constraints:
• Should be suitable for application of deep learning.
• The project must be of a suitable complexity and challenge that is similar to the
suggested projects. As part of the negotiation, the scope and deliverables of the
project will be set.
• The data set to be used in the project must be available without restrictions before
the start of the negotiation process.
• The proposed project must be independent of previously or concurrently assessed
work. You may not conduct a project if you have already been assessed on the
work, or are concurrently being assessed on the work.
In general, negotiations will take place via email, during consultation hours, or by
appointment. Please note, that the course co-ordinator is not available outside of business
hours.
All negotiated projects must be finalised by no later than 5pm Thursday Week
9 . This is the absolute deadline. If you wish to conduct a negotiated project, begin the
negotiation process early . A negotiated project may be denied before the deadline if
there is insufficient time for the negotiation process.
5 Submission
You have to submit all the relevant material as listed below via Canvas.
1. A report (of no more than 3 pages , plus up to 2 pages for appendices) critically
analysing your approach and ultimate judgement. Should be in PDF format.
2. Your code (Jupyter notebooks) used to perform your analysis. Should be a ZIP
file containing all the support files.
The submission portal on canvas consists of two sub-pages. page one for report
submission and the second page for code and other file submission. More information
is provided on canvas. Include only source code in a zip file containing your name. We
strongly recommend you to attach a README file with instructions on how to run your
application. Make sure that your assignment can run only with the code included in your
zip file! Include a PDF version of your report.
After the due date, you will have 5 business days to submit your assignment as a late
submission. Late submissions will incur a penalty of 10% per day. After these five days,
Canvas will be closed and you will lose ALL the assignment marks.
Assessment declaration:
When you submit work electronically, you agree to the assessment declaration – https://
www.rmit.edu.au/students/student-essentials/assessment-and-exams/assessment/
assessment-declaration
5
https://www.rmit.edu.au/students/student-essentials/assessment-and-exams/assessment/assessment-declaration
https://www.rmit.edu.au/students/student-essentials/assessment-and-exams/assessment/assessment-declaration
https://www.rmit.edu.au/students/student-essentials/assessment-and-exams/assessment/assessment-declaration
6 Academic integrity and plagiarism (standard warning)
Academic integrity is about honest presentation of your academic work. It means ac-
knowledging the work of others while developing your own insights, knowledge and ideas.
You should take extreme care that you have:
• Acknowledged words, data, diagrams, models, frameworks and/or ideas of others
you have quoted (i.e. directly copied), summarised, paraphrased, discussed or men-
tioned in your assessment through the appropriate referencing methods
• Provided a reference list of the publication details so your reader can locate the
source if necessary. This includes material taken from Internet sites. If you do not
acknowledge the sources of your material, you may be accused of plagiarism because
you have passed off the work and ideas of another person without appropriate
referencing, as if they were your own.
RMIT University treats plagiarism as a very serious offence constituting misconduct.
Plagiarism covers a variety of inappropriate behaviours, including:
• Failure to properly document a source
• Copyright material from the internet or databases
• Collusion between students
For further information on our policies and procedures, please refer to the following:
https://www.rmit.edu.au/students/student-essentials/rights-and-responsibilities/
academic-integrity.
7 Marking guidelines
A detailed rubric is attached on canvas. In summary:
• Literature Review 20%
• Approach 50%;
• Ultimate Judgment & Analysis 20%;
• Report & Code 10%;
Literature review: A through literature review should be conducted to identify the
state-of-the-art DL techniques relevant to the problem. A thorough investigation must
include evidence to justify your proposed methodology.
A good literature review:
• Should follow a logical structure.
• Should not just provide a list of related papers. Papers should be discussed in
relation to why are they relevant for this problem, what is good about them and
what are the limitations.
• Papers should be summarised in your own words. Directly copying sentences
from the reviewed paper is considered as plagiarism and significant penal-
ties will apply.
6
https://www.rmit.edu.au/students/student-essentials/rights-and-responsibilities/academic-integrity
https://www.rmit.edu.au/students/student-essentials/rights-and-responsibilities/academic-integrity
• Will discuss literature from peer-reviewed sources. Wikipedia or web discussion
forums are not considered as peer-reviewed sources.
Approach: You are required to use a suitable deep learning based approach to solve
the problem. Each element of the approach need to be justified using data analysis,
performance analysis and/or published work in literature. This assignment isn’t just
about your code or model, but the thought process behind your work. The elements of
your approach may include:
• Setting up the evaluation framework
• Selecting CNN architecture, loss function and optimization procedure.
• Hyper-parameter setting and tuning
• Identify problem specific issues/properties and solutions
• Demonstrate your skills on advanced concepts in deep learning.
Ultimate Judgement: You must make an ultimate judgement of the “best” model that
you would use and recommend in a real-world setting for this problem. It is up to you to
determine the criteria by which you evaluate your model and determine what is means to
be “the best model”. You need to provide evidence to support your ultimate judgement
and discuss limitation of your approach/ultimate model if there are any. You should use
independent test-data to conduct this investigation (may not apply to some negotiated
projects).
Critical Analysis & Report: Finally, you must compile a report describing and
analysing the approach that you have taken to find a suitable model and make your
ultimate judgement. Your report must be no longer that 3 pages , plus an additional
2 pages for appendices. The appendices must only contain references, figure, diagram,
or data tables that provide evidence to support the conclusions and statements in your
report.
Any over length content, or content outside of these requirements will not
be marked. For example, if you report is too long, ONLY the first 3 pages pages of text
will be read and marked.
In this report you should describe elements such as:
• Your final selected approach
• Why you selected this approach
• Parameter settings and other approaches you have tried.
• Limitation and improvements that are required for real-world implantation.
This will allow us to understand your rationale. We encourage you to explore this
problem and not just focus on maximising a single performance metric. By the end of
your report, we should be convinced about your ultimate judgement and that you have
considered all reasonable aspects in investigating this problem.
Remember that good analysis provides factual statements, evidence and justifications
for conclusions that you draw. A statements such as:
7
“I did xyz because I felt that it was good”
is not analysis. This is an unjustified opinion. Instead, you should aim for statements
such as:
“I did xyz because it is more efficient. It is more efficient because . . . ”
8
Overview
Learning Outcomes
Assessment details
Task
Suggested Projects
Determining Support for Rumours
Inferring Distance in Images
Negotiated Project
Submission
Academic integrity and plagiarism (standard warning)
Marking guidelines