INFS5710 IT Infra. for BA
INFS5710 Week 1
Copyright © 2012, SAS Institute Inc. All rights reserved.
Copyright By PowCoder代写 加微信 powcoder
Database Systems Infrastructure
Entity Relationship Model (ERM)
Normalisation
Prepared by , Feb. 2021
External Data (e.g. Excel)
ETL (Data Cleansing)
Star Schema (De-Normalised)
Data Warehouse
External Data (e.g. Excel)
Data (DW to BD, or vice versa or both)
Unstructured Data (Social Media)
Data Streaming
Structured Data (Internet of Things (IOT))
Data Streaming
Flat Files
Machine Learning
Relational Database
Today, we will cover the last bit you will learn in this course on Big Data. Now, you see how everything in the diagram is connected.
In summary, you have learned normalisation design of the database in Week 4 using ERD, i.e. Entity-Relationship Diagram, which you explored in Week 2 and 3. You learned to use Oracle to create an ERD in the lab.
Moreover, you have been learning SQL or Sequel in the Lab workshops.
You learned about data warehouse.
Again, there are extra materials for this lecture – all the text should be on slides.
This week, we will talk about Hadoop, MapReduce and NoSQL behind the building of big data. Also, on how data can be retrieved from relational database or saved the data back to the relational database. A data warehouse could be a relational database.
(“Not Normalised”)
Spark and NoSQL (and other tools)
Note: In-Memory Database (e.g. SAP Hana) is an alternative data model not shown here
Hadoop Distributed File System (HDFS) and MapReduce
Reporting (Business Intelligence and Visualisation) and Business Analysis (End Users)
The Human Face of Big Data
The impact of the Big Data could be described the next major revolution since the Agricultural Revolution and Industrial Revolution. We can call it Digital Revolution or Big Data Revolution. Today, we have already seen large corporations, particularly the large Chinese companies, use Big Data, Artificial Intelligence, and machine learning extensively to drive their business strategies to gain competitiveness.
This awarded winning documentary was created to explain how Big Data has evolved the way we work, shop, socialise, live, and benefit from Big Data as well as the rise of negative issues associated with Big Data. Big Data is collected, stored, and used across a wide range of products and services. You will learn how Big Data can be used in various areas, and how Big Data influence
©2017 Cengage Learning®. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website or school-approved learning management system for classroom use.
Next, this is a very documentary talking about Big Data, after watching this video, you virtually will learn everything about Big Data.
It summarises every topics about Big Data.
Video: The Human Face of Big Data
Ref: https://www.youtube.com/watch?v=bIY3LUZ7i8Y or
https://www.bilibili.com/video/BV1uE411t7CZ/
Warning:Themusicinthevideoisabitloudinsomesections,soyou
might want to test and control the volume.
©2017 Cengage Learning®. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website or school-approved learning management system for classroom use.
At home, before clicking the video, go and get some popcorn, crisp, candy and ice-pop, now you can click to watch the video on YouTube
Topic: Digitising Ourselves (17:36: 23:55)
Collecting data about oneself!
Pattern recognition algorithm – change the way as a
o Personal devices, such as , Samsung Watch, and Fitbit, contain apps and sensors used to collect data about your health (as an example).
oIf you have such personal devices, the question here is can these devices influence on how you behave. Examples can include do you pay attention to the output (such as graph or numbers) from these apps, or do you have a goal of burning number of
calories per day.
©2017 Cengage Learning®. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a
certain product or service or otherwise on a password-protected website or school-approved learning management system for classroom use.
As stated on the slide…
Just a thought: Think of devices contain sensors including IoT (Internet of Things) which might the change the way you behave (or not). For example, taking a photo with location enable using an iPhone, the photos can be now be sorted by date and location, you might use the photos as evidence that you have attended a conference for the purpose of tax deduction.
Topic: Building a Global Brain (23:55 to 25:55) Topic: Creating Intelligence System (25:55 to 28:50)
Data is collected from you via devices. You react based on the data presented to you, and the action you have taken becomes another data point in this Big Data system. This becomes a cycle where the Big Data has an impact on you, and then your action becomes a datapoint in the Big Data.
In the video, it discusses about scheduling of buses. One of the suggestions is to be more proactive based on the needs of bus, i.e. instead of ten buses regularly travelling on one route. The bus can be diverted to another route if the demand for this particular route is reduced but a higher demand for the other route. Some would call this as building a smart city from Big Data. Thus, the city like Boston could be functioned more efficiently based on the data, i.e. “responsive to our needs”
©2017 Cengage Learning®. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website or school-approved learning management system for classroom use.
As stated on the slide…
Something for you to think about…
Topic: Mapping the Cost of Justice (28:51 to 31:50) [1]
In this section of the video, the key point is “not looking at tons and tons of data, but what is the story behind it”, that is you must find a story behind the data.
In the video, the map above shows where the ex- This shows another map where African-America prisoners reside. reside. Furthermore, there is also a high
concentrate area of poverty.
©2017 Cengage Learning®. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website or school-approved learning management system for classroom use.
This is related directly to your assignment 2 – “how can you put one and one together?” You need to tell a story based on the data.
This is like the rubric cube, when you move them around, they look different. So, you have moved the rubric around, so it tells a different story.
On the left, it shows where the ex-prisoners live. On the right slide, it shows where the African –America live.
Topic: Mapping the Cost of Justice (28:51 to 31:50) [2]
If you put both maps together, you will see a high correlation between the ex-prisoners and where they live. Moreover, this creates “a million-dollar block”, which was the amount of money spent on people going to prison and then released back to the neighbourhood. Thus, money could be better spent dealing with this area.
In summary, although we say, “a picture is worth a thousand words”, but one picture might not paint the whole picture, as to speak, so you need a few pictures combine to enrich and create an overall picture.
©2017 Cengage Learning®. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website or school-approved learning management system for classroom use.
When you put the two pictures together, you get something is more useful, can be used to support and justify your argument, clean up the area and create a community. Thus, you need to move the money of keeping someone in jail to something that a community can use.
Although we say, “a picture is worth a thousand words”, but one picture might not paint the whole picture, as to speak, so you need a few pictures combine to enrich and create an overall picture. So in your assignment, you can think about how you can put things together – this actually is system thinking!
Topic: Targeting You (38:23 to 41:05) [1]
Target has used Big Data to identify pregnant women as part of their marketing strategy to target that segment of the consumers, provide better customer services, and improve their revenue. This practice is common among the retailers, hotel industry, airline industry and gambling industry, which offering loyalty program to their customers as a way of rewarding them for being their customers. The original intention of offering loyalty program is to build a customer relationship. However, in the case of Target, they use the customer information further with Big Data to create a profile of their customers who purchase products related to pregnancy and baby.
©2017 Cengage Learning®. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website or school-approved learning management system for classroom use.
As stated on the slide…
A classic story of using predicting data
Topic: Targeting You (38:23 to 41:05) [2]
Another example is nearly all the search engines, such as Google, generate their revenue by producing advertisements based on what your searches. Companies want to advertise their products on the Internet, and these search engine companies offer their services to the customers who search terms or phrases which meet the advertising criteria.
©2017 Cengage Learning®. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website or school-approved learning management system for classroom use.
As stated on the slide…
developed an algorithm based on data from signing up, buying patterns, and you can predict what will happen next.
Like the book example… This is how Google makes money.
Topic: The Dark Side (41:06 to 45:59)
One of the criticisms on Facebook is they have been collecting data without fully reveal their intention, and how they would use your data once they collected. They can build a profile of you as an individual. Moreover, National Security Agency (NSA) has been collecting data for a number of years.
©2017 Cengage Learning®. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website or school-approved learning management system for classroom use.
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com