程序代写代做 hadoop AWS distributed system file system algorithm database COMP5349 – Cloud Computing

COMP5349 – Cloud Computing
Week 13: Course Review and Exam Info
Dr. Ying Zhou School of Computer Science

COMP5349 Schedule in 2019
Week
Week 1
Week 2
Week 3
Week 4
Week 5
Topic
Cloud Computing Overview and Service Models
Virtualization Technology
Container Technology
Map/Reduce Framework
Spark Framework
Week 6
Week 7
Distributed Execution: HDFS and YARN
Distributed Execution: Spark
Week 8
Spark Data Frame
Week 9
Week 10
Week 11
Week 12
Week 13
Spark Machine Learning Library
Cloud Storage and Databases Services
Consistency in Cloud Storage and Database Service
Course Review
COMP5349 “Cloud Computing” – 2020 (Y. Zhou) 13-2

The Big Picture
n Cloud Computing
„ Shared IT services for clients to rent from
„ On different levels (IaaS, PaaS, SaaS, FaaS, ….)
„ Made possible through web and data center technology
n Enabling Technologies „ Virtualization
¡ Used by all IaaS providers „ Container
¡ Can be used in various scenarios
• IaaS customers can use container technology to deploy applications on VM
• PaaS or SaaS
¡ Active development to support Serverless computing
„ Key issues
¡ Illusion of a whole system to every client ¡ Performance isolation
¡ Security and others
COMP5349 “Cloud Computing” – 2020 (Y. Zhou) 13-3

Analytics and BigData Services
n Basic Computational Model
„ Storage: distributed file systems (GFS, HDFS)
„ Programming Paradigm: MapReduce
„ Hadoop MapReduce as specific (open source) example „ Map and Reduce phases
¡ Each phase allows multiple tasks to run in parallel
¡ Synchronization and shuffling happen between map and reduce phase
¡ Map output key is used to reorganize intermediate result in reduce phase
„ An analytic workload may needs several map reduce phases
„ Simple localized fault tolerance mechanism depends on storage and I/O n Other computation model
„ Spark
¡ RDD based API
¡ Data Frame based API
„ Main-memory based as compared with disk/batch based approach by MapReduce
n All based on functional programming paradigm
COMP5349 “Cloud Computing” – 2020 (Y. Zhou) 13-4

Cloud Storage and Database Services
n Cloud Storage Services
„The cloud version of file system: GFS/HDFS, S3, EBS, etc
n Cloud Database Services
„The cloud version of database: Bigtable, WAS, Dynamo, AWS
Aurora
n Common features „ Replication
„ Partition
„Fault Tolerance
„Various consistency levels
„Various ways of handling read/write of the data
COMP5349 “Cloud Computing” – 2020 (Y. Zhou) 13-5

Cloud Storage/DB Services Consistency
n Many systems use customized algorithms for handling read/write
n Classic distributed system algorithm „ Paxos
¡ First phase only requests participants to make a promise, the actual value is proposed in the second phase
¡ Participant is not requested to check the value proposed, instead, participant checks the proposal’s sequence number
¡ A leader is necessary to maintain the progress of the algorithm
n Paxos can be used in replicated environment to reach
consensus
„ Run multiple Paxos, each is numbered and the value to be chosen represents an update command
„ Efficient mechanism to run infinite Paxos
COMP5349 “Cloud Computing” – 2020 (Y. Zhou) 13-6

Final Exam
n Online Open book two-hour exam „Conducted in Canvas
n Exam questions
„Multiple choice questions and short answer questions
n Multiple choice question has single correct answer
n Short answer questions are structured into multiple parts.
n Type your answers in the text field provided in the exam paper
„Label answer with question part
n The exam has a 100 points in total
n The exam has a 40% barrier
„You need to get at least 40 of 100 points in the final exam
to pass this subject
COMP5349 “Cloud Computing” – 2020 (Y. Zhou) 13-7

Final Exam Content
n Assessable: „Lecture content
¡ All weeks „Tutorial material
¡ All except week 1 „Assignment
n Big Data programming may be assessed in various ways „Questions based on a short program
„Design a workload by writing code/pseudo code
COMP5349 “Cloud Computing” – 2020 (Y. Zhou) 13-8

Thank You!
COMP5349 “Cloud Computing” – 2020 (Y. Zhou) 13-9