COMP3430 / COMP8430 Data wrangling
– We will start 5 minutes past the hour: 13:05 AEDT or thereabouts.
Interactive lecture Week 10: Admin, Summary of Ontology, Temporal, Spatial (Lecturer: )
Lecture outline
● Administrative matters
– Feedback, Assignments, Labs, Exam, Lectures
● Question from Wattle ● Summary of Week 10:
Ontologies, Big, Temporal, Spatial Data ● Q and A Session
Feedback is Important
● Course representatives have published a Feedback Form ● See the announcement in EITHER forum
● What worked, not worked, ideas for better experience, etc.
Labs
● Lab 7 this week
Python linkage and Linkage Evaluation
● Optional Lab next week to reinforce your learnings Privacy Preserving Record Linkage
Assignments
● Assignment 3 due Friday 22 October 23:55 (AEDT)
● Please SUBMIT properly ● Iterate:
● Finish attempt
● Return to attempt
● Finalise (once only before the deadline):
● Finish all and submit
From Wattle
Upload your output file which contains the linked and classified matching record pairs for the best linkage result you were able to obtain.
Your output file must exactly follow the CSV file format which was generated using the function save_linkage_set() in the Python
program saveLinkResult.py which we use in lab 7.
Each row in this file must contain two record identifiers only.
You must name the file you upload
as: data_wrangling_rl_best_results_2021_your_ANU_ID.csv, where you replace your_ANU_ID with your actual ANU ID, for example data_wrangling_rl_best_results_2021_u1234567.csv
DO NOT enter any text into the text field below. DO NOT provide any other files such as python scripts or screenshots.
Assignments
● Assignment 4 – COMP8430 students only – due Friday 29 October at 23:55 (AEDT)
● Before you post a question on Wattle please read all previous posts!
Final examination
● Final exam will be on Monday 15th November in the afternoon from 5.40 PM AEDT.
● For details see ANU exam timetable.
● We recommend you to have Python and Rattle available
through, for example, VirtualBox or VMWare Horizon.
● We will discuss final exam in the last interactive lecture.
Lectures
● This week marks the end of the video lectures
● No interactive lecture next week – focus Assignments 3/4
● Final interactive lecture Friday 29 October 1pm
● Course summary and examination information.
Questions from Wattle forum
● Q1: Concrete Example of Trend Discovery
Topic: Ontologies
● Humans create ontologies to better understand the world. ● Animal World and Plant World and Manufactured World
● As we learn more the ontologies evolve
: α β γ δ ε ζ […] – the ontology is growing
● As ontologies evolve we add, remove, modify concepts
● Trends over time can help understand where we might be
heading: new variations are added exponentially more quickly? We capture knowledge and our knowledge changes over time.
●
●
Topic: Big Data
● We had an “era” of big data – 2005 – 2017 (roughly) ● Today’s era of AI is Big Data and Big Compute
● The 4 Vs: Volume, Variety, Velocity, Veracity
● Compare Amazon 1996 and 2021
Topic: Temporal (and Dynamic) Data
● Data captured over time often at Velocity – single view of records
● Wrangling needs to deal with changing ontologies: concepts, distributions, ranges, outliers, changing over time – are they the same concept or different, emerging concept or outlier, etc?
● MACHOs – the missing dark matter of the universe ● Micro-lensing effects
● Weather, Locations, Salaries, Age, Tax Scales and Policy, …
Topic: Spatial (and Location) Data
● Location and relationships between adjacent objects
● Wrangling including anonymising data, inaccurate locations,
visualise for verification (open street map), infer missing locations
● COVID-19 (crisper.net.au) Exposure Sites
● Bushfires in Kakadu National Park
● Navigating the World with the “Help” of Google
● Privacy – our data and analysis back to the personal device
Q and A Session
● Socrative
– https://b.socrative.com/login/student/ – Room Name: COMP3430