Deep Learning COMP 5329
Dr Chang Xu
c.xu@sydney.edu.au
School of Computer Science
The University of Sydney
Page 1
AI’s Big Breakthroughs in 2020
GPT-3, the 3rd version of Generative Pre-Trained Transformer model released by OpenAI.
Develop Web apps. Enter a sentence
describing Google home page layout, and
here you see GPT-3 generating the code for
Building ML Models. Keras code written by GPT-3 while input was simple plain text of what ML model do we want to write code for, and boom, it generated the model. Page 2
it.
The University of Sydney
AI’s Big Breakthroughs in 2020
AlphaFold: a solution to a 50-year-old grand challenge in biology.
Proteins are very complex molecules that do all the work in a cell. In order to understand how proteins work, one needs to know what structure they adopt when they fold into their final conformation.
The University of Sydney
Page 3
Artificial Intelligence: Human Intelligence Exhibited by Machines.
Machine Learning : An Approach to Achieve Artificial Intelligence
Deep Learning: A Technique for Implementing Machine Learning
The University of Sydney
Page 4
What is Deep Learning?
The short answers
1. ‘Deep Learning’ means using a neural network
with several layers of nodes between input and output
2. the series of layers between input & output to do
feature identification and processing in a series of stages, just as our brains seem to.
The University of Sydney Page 5
What is Deep Learning?
We have always had good algorithms for learning the weights in networks with 1 hidden layer;
But these algorithms are not good at learning the weights for networks with more hidden layers
Image credit to becominghuman.ai
The University of Sydney Page 6
What to Learn?
https://canvas.sydney.edu.au/courses/30917/pages/course-contents
1. Introduction
2. Multilayer Neural Networks
3. Optimization for Deep Models
4. Regularization for Deep Models 5. Holiday
6. Convolutional Neural Networks 7. Neural Network Architectures
8. Recurrent Neural Networks
9. Transformer Neural Networks 10. Graph Convolutional Networks 11. Deep Learning Applications
12. Deep Generation Models
13. Review
The University of Sydney
Page 7
Enrollment of COMP5329
Enrollment of COMP5329 400
300 200 100
0
2018 2019 2020 2021
The University of Sydney
Page 8
Online Lectures and Tutorials
– –
Lecture: Friday 18:00 – 20:00
Tutorial: Thursday 17:00 – 18:00; Friday 20:00 – 21:00 (from the 2nd week)
–
–
Consultation: Friday, 15:00-16:00 https://uni-sydney.zoom.us/j/7228816221?pwd=RmQ3dEgzY25ETXRrSkh6ekVBSyt6Zz09
Password: 135109
Teaching Assistant: Mr Gary Jiajun Huang
Ed Discussion
https://edstem.org/courses/5205/discussion/385710
The University of Sydney Page 10
COMP5329 Reference book
Goodfellow I J, Bengio Y, Courville A, Deep Learning. MIT Press, 2016.
LeCun Y, Bengio Y, Hinton G. Deep learning[J]. nature, 2015, 521(7553): 436-444.
• We will use PyTorch in labs. PyTorch documentation can be found at https://pytorch.org/docs/stable/index.html
The University of Sydney Page 11
COMP5329 Reference book
Applied Math and Machine Learning Basics:
Goodfellow I J, Bengio Y, Courville A, Deep Learning. MIT Press, 2016.
The University of Sydney Page 12
Assessment Overview
– Assignment1:20%
– Week-7 (23-April) 18:00
– Groups of 2 or 3 students
– Classification task
– Assignment2:20%
– Week-12 (28-May) 18:00
– Groups of 2 or 3 students for Competition Track
– Groupsof2or3studentsforResearchTrack
– Finalexam:60%
– date to be defined
The University of Sydney
Page 13
Call for research track abstracts for Assignment 2
(see Canvas)
Ø This research track calls for brave new ideas on deep learning. In this track, you are encouraged to propose and investigate new algorithms or problems in deep learning.
Ø You must not use the project (e.g. your capstone or SSP project) that you have already done or are currently doing in other units to participate in this track.
Ø An extended abstract (up to one page) to highlight the importance and novelty of your research problem, the major idea of your candidate solution, the feasibility of the proposed research, and the expected experimental achievements (if applicable).
Ø 10 groups will be selected.
Ø Abstract Submission Deadline: Week 4 (26 March)
Ø Abstract Decision: Week 6 (16 April)
The University of Sydney Page 14
Expectations
– Students attend scheduled classes, and devote an extra 6-9 hrs per week
– doing assessments
– preparing and reviewing for classes
– revising and integrating the ideas
– practice and self-assess
– Studentsareresponsiblelearners
– Participate in classes, constructively
• Respect for one another (criticize ideas, not people)
• Humility: none of us knows it all; each of us knows valuable things
– CheckCanvassiteatleastonceaweek!
– Notify academics whenever there are difficulties
– Notify group partners honestly and promptly about difficulties
The University of Sydney Page 15
Special Consideration (University policy)
– If your performance on assessments is affected by illness or misadventure
– Follow proper bureaucratic procedures
– Have professional practitioner sign special USyd form
– Submit application for special consideration online, upload scans – Note you have only a quite short deadline for applying
– http://sydney.edu.au/current_students/special_consideration/
– Also, notify coordinator by email as soon as anything begins to go wrong
– There is a similar process if you need special arrangements eg for religious observance, military service, representative sports
The University of Sydney Page 16
Late assessments in COMP5329
– Supposeyouhandinworkafterthedeadline:
– Ifyouhavenotbeengrantedspecialconsiderationor
arrangements
– Apenaltyof5%ofthemaximummarkswillbetakenperday(orpart) late. After ten days, you will be awarded a mark of zero.
– e.g.Ifanassignmentisworth40%ofthefinalmarkandyouare one hour late submitting, then the maximum marks possible would be 38%.
– e.g.Ifanassignmentisworth40%ofthefinalmarkandyouare 28 hours late submitting, then the maximum marks possible marks would be 36%.
– Warning:submissionsitesgetveryslowneardeadlines
– Submitearly;youcanresubmitifthereistimebeforethedeadline
The University of Sydney Page 17
Academic Integrity (University policy)
– “The University of Sydney is unequivocally opposed to, and intolerant of, plagiarism and academic dishonesty.
– Academic dishonesty means seeking to obtain or obtaining academic advantage for oneself or for others (including in the assessment or publication of work) by dishonest or unfair means.
– Plagiarism means presenting another person’s work as one’s own work by presenting, copying or reproducing it without appropriate acknowledgement of the source.” [from site below]
– http://sydney.edu.au/elearning/student/EI/index.shtml
– Submitted work is compared against other work (from students, the
internet, etc)
– TurnitIn for textual tasks (through Canvas), other systems for code
– Penalties for academic dishonesty or plagiarism can be severe
– Complete self-education AHEM1001 (required to pass INFOxxxx)
The University of Sydney
Page 18
COMP5329: Resources
– Canvas:https://canvas.sydney.edu.au/
– Login using Unikey and password
– Link to Units website: https://sydney.edu.au/units/
• Official schedule, list of learning outcomes, etc – Copies of slides
– Lab instructions
– Assignment instructions
– Lecture videos
– Submit official assignment work here;
– see your grades; etc
– Discussion forum: on edstem The University of Sydney
Page 19
WHS Induction
School of Computer Science
The University of Sydney Page 20
General Housekeeping – Use of Labs
– Keep work area clean and orderly
– Remove trip hazards around desk area
– No food and drink near machines
– No smoking permitted within University buildings
– Do not unplug or move equipment without permission
The University of Sydney
Page 21
EMERGENCIES – Be prepared https://sydney.edu.au/campus-life/safety-security.html
The University of Sydney Page 22
EMERGENCIES
Assembly Area: Cadigal Green
The University of Sydney
Page 23
EMERGENCIES
The University of Sydney
Page 24
l
– If a person is seriously ill/injured:
MEDICAL EMERGENCY
1. call an ambulance 0-000
2. notify the closest Nominated First Aid Officer
If unconscious– send for Automated External Defibrillator (AED) AED locations.
NEAREST to CS Building (J12)
– Electrical Engineering Building, L2 (ground) near lifts
– Seymour Centre, left of box office
– Carried by all Security Patrol vehicles
3. call Security – 9351-3333
4. Facilitate the arrival of Ambulance Staff (via Security)
Nearest Medical Facility
University Health Service in Level 3, Wentworth Building
First Aid kit – SIT Building (J12) kitchen area adjacent to Lab 110
The University of Sydney
Page 25
School of Computer Science Safety Contacts
CHIEF WARDEN Greg Ryan
Level 1W 103 9351 4360 0411 406 322
FIRST AID OFFICERS
Julia Ashworth Level 2E Reception
9351 3423
Will Calleja Level 1W 103
9036 9706 0422 001 964
Katie Yang Level 2E 237
9351 4918
Undergraduates:
Coursework Postgraduates:
CS School Manager:
OR
to Katie Yang 9351 4918
to Cecille Faraizi 9351 6060
or Keiko Narushima 8627 0872
Priyanka Magotra 8627 4295
The University of Sydney
Page 26
Assistance
– There are a wide range of support services available for students: https://sydney.edu.au/campus-life/health- wellbeing-success.html
– Please make contact, and get help
– You are not required to tell anyone else about this
– If you are willing to inform the unit coordinator, they may be able to work with other support to reduce the impact on this unit
– eg provide advice on which tasks are most significant
The University of Sydney Page 27
DISABILITY SERVICES
Do you have a disability?
– You may not think of yourself as having a ‘disability’ but the definition under the Disability Discrimination Act is broad and includes temporary or chronic medical conditions, physical or sensory disabilities, psychological conditions and learning disabilities.
– The types of disabilities we see include:
– anxiety, arthritis, asthma, asperger’s disorder, ADHD, bipolar disorder, broken bones, cancer, cerebral palsy, chronic fatigue syndrome, crohn’s disease, cystic fibrosis, depression, diabetes, dyslexia, epilepsy, hearing
impairment, learning disability, mobility impairment, multiple sclerosis, post traumatic stress, schizophrenia , vision impairment, and much more.
– Students needing assistance must register with Disability Services – – it is advisable to do this as early as possible.
– http://sydney.edu.au/study/academic-support/disability-support.html
The University of Sydney Page 28
Do you have a disability?
You may not think of yourself as having a ‘disability’ but the definition under the Disability Discrimination Act (1992) is broad and includes temporary or chronic medical conditions, physical or sensory disabilities, psychological conditions and learning disabilities.
The types of disabilities we see include:
Anxiety // Arthritis // Asthma // Autism // ADHD Bipolar disorder // Broken bones // Cancer
Cerebral palsy // Chronic fatigue syndrome Crohn’s disease // Cystic fibrosis // Depression Diabetes // Dyslexia // Epilepsy // Hearing impairment //
Learning disability // Mobility impairment // Multiple sclerosis // Post-traumatic stress // Schizophrenia // Vision impairment
and much more.
Students needing assistance must register with Disability Services. It is advisable to do this as early as
possible. Please contact us or review our website to find out more.
The University of Sydney
Disability Services Office sydney.edu.au/disability 02-8627-8422
Page 29
Other support
– Learning support
– http://sydney.edu.au/study/academic-support/learning-support.html
– International students
– http://sydney.edu.au/study/academic-support/support-for-international-students.html
– Aboriginal and Torres Strait Islanders
– http://sydney.edu.au/study/academic-support/aboriginal-and-torres-strait-islander-
support.html
– Student organization (can represent you in academic appeals etc)
– http://srcusyd.net.au/ or http://www.supra.net.au/
– Please make contact, and get help
– You are not required to tell anyone else about this
– If you are willing to inform the unit coordinator, they may be able to work with other support to reduce the impact on this unit
– eg provide advice on which tasks are most significant
The University of Sydney Page 30
Advice
– Metacognition
– Pay attention to the learning outcomes in Canvas
– –
– Self-check that you are achieving each one
– Think how each assessment task relates to these
Time management
– Watch the due dates
– Start work early, submit early
Networking and community-formation
– Make friends and discuss ideas with them
– Know your tutor, lecturer, coordinator
– Keep them informed, especially if you fall behind
• Don’t wait to get help Enjoy the learning!
–
The University of Sydney
Page 31
Invited talks
– Xikun Zhang – “On Dropping Clusters to Regularize Graph Convolutional Neural Networks”
– KwonNungChoi-“VideoTrain:AGenerativeAdversarial Framework for Synthetic Video Traffic Generation”
– Xinqi Zhu – “Learning Disentangled Representations with Latent Variation Predictability”
– Yanxi Li – “Adapting Neural Architectures Between Domains”
The University of Sydney Page 32