Cluster and Cloud Computing – Lecture 1 & 2 Professor Richard O. Sinnott
Director, Melbourne eResearch Group University of Melbourne
Copyright By PowCoder代写 加微信 powcoder
Director, eResearch University of Melbourne
Chair in Applied Computing Systems, University of Melbourne
BSc Theoretical Physics
Technical Director National e- Science Centre, University of Glasgow
CEO Own Company (real time systems/telecoms)
PhD Distributed Systems
Multiple PhD/MSc supervised in this area
Director, Bioinformatics Research Centre University of Glasgow
MSc Software Engineering
Post-doc GMD Fokus Berlin
Distributed Systems Standards creator
MSc Statistics
IS Software Engineer, ESRI
Data Architect, AURIN Project
Committer at the Apache Software Foundation
IT Consultant for the World Bank, United Nations, European Union, Asian Development Bank
University of Melbourne Doctor of Philosophy (PhD)
Cloud Computing Container and Orchestration tool benchmarking
devOps (SWARM, repliCATS, HATS)
COMP90024 graduate
Master of Science IT UniMelb
Melbourne eResearch Group (2022) (http://eresearch.unimelb.edu.au)
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Funding (>$300m) from many funding sources (DARPA, IARPA, DSTG, EU, JDRF, HCT, NHMRC, ARC, Dept Innovation, Dept Environment, DELWP, VicHealth, Commercial, …)
800+ Masters dissertations
ALL ABOUT APPLIED COMPUTING (typically data and security!!!)
Course Contents
• Lectures 1 & 2 – 2nd March (workshops 8th March)
– Information Session & How we got here (Distributed Systems, Grid…) •
• Workshop #1/#2/#3 (Q&A on the lectures)
• Lectures 3 & 4 – 9th March (workshops 15th March)
– Domain Drivers – tour of some big data projects •
• Workshop #1 (Q&A on the lectures)
• Workshop #2/#3 demo on driving AURIN (needed for assignment 2)
• Lectures 5 & 6 – 16th March (workshops 22nd March)
– Parallel Systems, Distributed Computing and HPC/HTC •
• Workshop#1 (Q&A on the lectures) • Workshop #2/#3 on Git ( )
First assignment handed out – 16th March
(Warning SPARTAN & volume students!!! Team of two for assignment!!!)
Course Contents
• Lectures 7 & 8 – 23rd March (workshops 29th March)
– HPC @ UniMelb and Practicalities of HPC/HTC
• Linux / HPC practicalities and welcome to Spartan!!! • Workshop#1 (Q&A on the lectures)
• Workshop #2/#3 on using SPARTAN & using mpi4py
• Lectures 9 & 10 – 30th March (workshops 5th April)
– Cloud Computing – Programming Clouds: Getting to grips with the UniMelb Research Cloud!
• Introduction to Cloud Computing
• Getting to grips with OpenStack/UniMelb Research Cloud
• Workshop#1 (Q&A on the lectures)
• Workshop #2/#3 on Scripting the Cloud (Introduction to Ansible demonstration)
Course Contents…ctd
• Lectures 11 & 12 – 6th April (workshops 12th April)
– ReST, Twitter (Needed for Assignment II) & Docker
• & & Jinyoung Kim
• Web services and Representational State Transfer (ReST)
• Examples of coding/demonstrating ReST and Twitter (Jinyoung)
• Introduction to Containers ( )
• Workshop#1 (Q&A on the lectures)
• Workshop #2/#3 on Demonstration of Docker/Docker SWARM/Kubernetes ( )
First assignment due in on 6th Programming Assignment handed out 6th April
Course Contents… ctd
• Lectures 13 & 14 – 13th April (“workshops 14th April”)
– Big Data and Related Technologies
• (Data Architect, AURIN)
• Big Data V-challenges, CAP Theorem and noSQL technologies • Workshop#1 (Q&A on the lectures)
• Workshop #2/#3 on CouchDB via Docker (Luca)
Easter Break (15-24th April)
• Lectures 15 & 16 – 27th April (workshops 3rd May)
– Cloud Underpinnings and Other Things
• Virtualisation background (Rich)
• Compare and Contrast AWS with NeCTAR ( )
• Workshop#1 (Q&A on the lectures)
• Workshop #2/#3 on serverless architectures and demonstration of Fn (Luca)
• Lectures 17 & 18 – 4th May (workshops 10th May)
– Big Data Analytics
• Big Data Technologies – Hadoop, HDFS, Spark, …
• Workshop#1 (Q&A on the lectures)
• Workshop #2/#3 on Hadoop cluster on Cloud (Luca)
Course Contents… ctd
Second Programming Assignment due in 10th May
• Lecture 19 & 20 (and workshop slots) – 10/11th May
– You (+ pizzas for those on campus…)
• ~20 teams randomly chosen to present their assignment II • 15 minutes each
• Lecture 21 & 22 (and workshop #2/#3 slots) – 18th May
– Security and Clouds & demonstration of assignments • & You
• 8 more teams randomly chosen to present their assignment II • 15minutes each
• Workshop #1 (Q&A on the lectures)
• Lecture 23 & 24 – 25th May
– Subject Review and Working Through Past Papers
– Feedback and SES •
At the end of the course….
• You will…
• Understand more about the history of cluster and Cloud computing and the current state of the art
• Know more of the domain drivers that are shaping this area – especially the current flavour of the month “big data”
• Understand more on parallel systems, multi-core software development
• Be able to use HPC/HTC systems
• Be able to use Cloud resources
• Be able to develop applications running on the Cloud
• Learning about next generation data management systems
• Be savvy with web based systems development
• Work on a non-trivial software system development as part of a team (=how it nearly always works in the real world!)
• Learn more about the pulse of cities!
• Have software experiences that are in great demand
Many of my ex-students have jobs in this space (reference machine!!!)
• Have visited a data centre
(for those in Australia!)
Lecture Slides
• On the web before lecture
• They may be updated slightly before the lecture to reflect recent developments
– Or cover materials that folk want more details on from previous lectures
• Other sources of materials – Key papers
– Interesting articles
– Snippets from the web
– Crib sheets for Cloud Computing – Crib sheets for HPC
Course Assessment
• Assignments
– During semester worth 50%
– Expected to take about 40+ hours
• Written examination
– A written examination (two hours) at the end of the semester worth 50%
– It will be online!!!
• All components must be completed satisfactorily to pass the subject
– At least 50% in all assignments + exam
Assessments
• (Mini Team) Assignment 1 – 10 marks
– HPC exercise (big(gish) data processing)
• (Bigger Team) Assignment (40 marks) – Social media analytics on the Cloud
– Team-based approach
– Self organise vs be organised • Importance of being a team player! • Peer review…
• Online Exam – 50 marks
– Importance of original work
• Collusion “seriously frowned” upon
• Note several students last year got 0 for 1st assignment
for not listening to this!!!
Computational Resources
• HPC systems
– SPARTAN cluster – general purpose cluster for UniMelb • Provisioned last year
• Reasonable grunt, but… UniMelb!
• Warning!!!
• Cloud resources
– UniMelb Research Cloud
• (https://dashboard.cloud.unimelb.edu.au/)
• UniMelb resources specifically set aside for course!
Workshops / Lab Sessions…?
• In this course workshops (#2/#3) are really demonstrations
– Materials made available before the workshop for you to follow during workshop (or at your own time at a later date)
– We will teach the theory and then do live demonstrations of solutions
– There are no hands-on/guided lab sessions • International students!?
– Any/all questions/discussions via the LMS • I live on Canvas for the next 3 months
– Workshop #1 is really a Q&A on the previous lectures (or indeed any aspect of the course!!!!)
• Targeted to those international students that may have questions that couldn’t be answered during the live stream lecture
Final Comments
• This is not an easy course!
• I assume that you are already reasonably savvy software engineers, but…
– You’ll be exposed to things that are in the sys-admin space
• Learning how to write scripts • Configure network settings
• Mount volumes
• Deploy systems
• You should be able to work things out yourselves
• MANY students get jobs in this space
– it is in the sweet spot of what industry wants/needs – Ex-students now working for Google, AWS, …
Questions?
The not so long ago buzz…
The latest buzz…
The cycles we go through…
The Hype Cycle… (Gartner 2015)
Not just IT trends…
Lessons…
• Don’t be fooled by the hype – Marketing and business drivers
• The basic principles of software engineering persist
– NeSC -> MeG experiences • Data processing,
• Distributed systems, • Security …
• Learning from experiences is key
– But obviously working on/with the bleeding edge can do no harm (for grants or on CVs)
Computing and Communication Technologies (r)evolution: 1960-…!
* Mainframes
* XEROXPARC
* TCP/IP * Ethernet
* Internet Era
* WSClusters
Centralised
* Minicomputers
* HTC * Workstations
* PDAs * P2P
* Grids * PCClusters
* Email * ARPANET
* Web Services
* XML * SocialNet
2000 2010 Decentralised
* e-Science
* e-Business
* e-Research
Centralised?
Cloud Computing IaaS, PaaS, SaaS
Communication
Cloud Computing….
• …is a colloquial expression used to describe a variety of different types of computing concepts that involve a large number of computers that are connected through a real-time communication network (typically the Internet). Cloud computing is a jargon term without a commonly accepted non-ambiguous scientific or technical definition. In science, cloud computing is a synonym for distributed computing over a network and means the ability to run a program on many connected computers at the same time. The popularity of the term can be attributed to its use in marketing to sell hosted services in the sense of application service provisioning that run client server software on a remote location.
Wikipedia 2013
Cloud Computing….
• …Proponents claim that cloud computing allows companies to avoid upfront infrastructure costs, and
focus on projects that differentiate their businesses instead of on infrastructure. Proponents also claim that cloud computing allows enterprises to get their applications up and running faster, with improved manageability and less maintenance, and enables IT to more rapidly adjust resources to meet fluctuating and unpredictable business demand. Cloud providers typically use a “pay as you go” model. This can lead to unexpectedly high charges if administrators do not adapt to the cloud pricing model.
Wikipedia 2016
Cloud Characteristics • Five essential characteristics:
– On-demand self-service. A consumer can provision computing capabilities as needed without requiring human interaction with each service provider.
– Networked access. Capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous client platforms.
– Resource pooling. The provider’s computing resources are pooled to serve multiple consumers using a multi-tenant model potentially with different physical and virtual resources that can be dynamically assigned and reassigned according to consumer demand.
– Rapid elasticity. Capabilities can be elastically provisioned and released, in some cases automatically, to scale rapidly upon demand.
– Measured service. Cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service.
– National Institute of Standards and Technology30
Cloud Computing….
• Flavours
– Compute clouds
• Amazon Elastic Compute Cloud • Azure
– Data clouds
• Amazon Simple Storage Service • Google docs
– Application clouds • App store
• Virtual image factories • Community-specific
– Private, public, hybrid, mobile, health, … clouds
• Edge, Fog, IoT, …
In the beginning….
• http://www.internetsociety.org/internet/what- internet/history-internet/brief-history-internet
Distributed Systems – A Very Brief History
• Once upon a time we had standards
– With very detailed conformance, consistency and compliance demands
• Services, protocols, inter-operability, …
• Then we had more standards
– Open distributed processing
– With slightly less rigorous compliance demands
• OMG Common Object Request Broker Architecture (CORBA)
• Distributed Computing Environment • Multiple technologies
– Client server, remote procedure call, …
Key distributed systems focus mid-90s
• Transparency and heterogeneity of computer- computer interactions
– finding/discovering resources (trader!), – binding to resources in real time,
– run time type checking,
– invoking resources
– dealing with heterogeneity of systems • applications and operating systems
– focused on computer-computer interaction
Challenges of earlier distributed (networked) system implementations
• Complexity of implementations – Middleware bloat and lock-in
– Vision and challenges of reality
• Vendor specific solutions • AnsaWare
• IBM Distributed SOM / Component Broker • Microsoft Network OLE
• IONA Orbix
• BEA ObjectBroker
• HP ORBPlus and Distributed SmallTalk
• Expersoft PowerBroker
• Other ORB’s
– Less mature standards
• Scale of the problem area
– Telecoms, banking, …
– The growth (boom!) of the web
Distributed Systems History…ctd
• Enter the web era
– My first ftp 1993 put/get files to/from Australia – Then the web pretty much exploded
• Peer-peer processing – File sharing …
• Scaling of…
– machines,
– domains of application
• Grid computing
– From computer-computer focus
– To organisation-organisation focus
e-Science and the Grid
infrastructure that will enable it.’
‘e will change the dynamic of the way science is undertaken.’
Director General of Research Councils
Office of Science and Technology
Grid is infrastructure used for e-Science
• Power Grid Metaphor: compute and data resources on demand
Major investment by UK Govt (£250m+) to realise this vision
-Across ALL Research Councils (+ EU + industry + …)
-Cyber-infrastructure, European Grid Initiative, Superscience, …
From presentation by
– Science Research
is about global collaboration in key areas of science, and the next generation of
Circa 2001
– Science Research
The Grid Metaphor
Overcoming heterogeneity…
Typical (Compute) Grid Architecture
All sorted…?
Grid Standards
• Even increasing numbers and ever decreasing understanding and acceptance/adoption by implementers
– Global Grid Forum – Open Grid Forum – OASIS
– W3C –…
+ many more…
Grid Technologies… (~2002+) • Globus Toolkit Project – www.globus.org
– GT2 – Complex software system for large, scale distributed software systems development
• Physiology of the Grid (Foster et al) www.globus.org/alliance/publications/papers/ogsa.pdf
– The Open Grid Services Architecture
• The Anatomy of the Grid: Enabling Scalable Virtual Organizations (Foster et al) http://dl.acm.org/citation.cfm?id=1080667
– MANY MB of source code
• Many software engineers worked in making this
– and many more in making it work!!!
Grid Technologies… (~2004+) • Move to service-based approach
• The Open Grid Services Infrastructure (OGSI)
• www.globus.org/toolkit/draft-ggf-ogsi- gridservice-33_2003-06-27.pdf
• GT3 – core technologies re- factored as “Grid Services”
• stateful Web services
• extension of Web services interfaces
• asynchronous notification of state change
• references to instances of services
• collections of service instances
• service state data augmenting constraints of XML Schema definition
Grid Technologies… (~2009+)
• Complete reassessment of OGSI approach to be “purer” web services
– Web service resource framework (WSRF) – GT4 implemented this
• Many software engineers hardened their skill sets using this (is one nice way of putting it!!!!) The only way to make software secure, reliable, and fast is to make it small
AS Tanenbaum • Also MANY other standards and efforts …
– Business and commercial drivers
• Vendors shaping standards to their commercial advantage
Flux of Web Service Standards (simplified!!!)
The nice thing about standards is that you have so many to choose from…
The development of robust Grid security infrastructures is very much dependent upon agreements on technologies and practices. Standardisation plays an extremely important role in this regard. With the move of the Grid community towards web services and service-oriented architectures, web service security standards and their associated implementations are crucial. Unfortunately it is the case that a multitude of specifications and proposals for web service standards have been promised and put forward, or merely promised. There are often cases of web service standards covering similar topics resulting in multiple competing specifications such as WS-Notifications and WS-Eventing; WS-ReliableMessaging and WS-Reliability; WS-Orchestration, WS-Co-ordination and WS-Choreography, along with the many varieties of workflow or business process languages that have been put forward to name but a few examples of the issues in the proliferation of web service standards. It is also the case that at the time of writing, many web services standards are only in working draft or draft status, often with no associated implementations or acknowledged conformance or interoperability definitions. Claiming conformance or compliance to a particular web service standard is thus often not possible (or meaningful!).
It is also apparent that although many standards use the common prefix “WS-”, this does not mean that there is an agreed WS-Architecture. This stems from a variety of reasons: vendor and commercial issues; political aspects and also the different bodies involved. For example the Internet Engineering Task Force (IETF) (www.ietf.org); the World Wide Web Consortium (W3C) (www.w3.org); the Organization for the Advancement of Structured Information Standards (OASIS) (www.oasis-open.org); and the Web Services Interoperability Organization (WS-I) (www.ws-i.org) are some of the most prominent bodies. The consequence of this profusion of standards and standards making bodies, and the lack of consensus on the core web service architecture, impacts directly upon development of Grid standards, architectures and associated implementations and middleware.
Me: Grid Security : Practices, Middleware and Outlook
The UK Grid Ecosystem
Circa 2004
National Centre for e-Social Science
National Centre for Text Mining
NERC e-Science Centre
National Data Centres
+ UK Federation
+ International dimension
including EGEE/EGI + SuperJanet
+ Training/Education +…
e-Science Institute
Core NGS Nodes +HECTOR +partners/affiliates (HECTOR investment £113m)
Grid Operations Support Centre
National Institute for Environmental e-Science
Digital Digital Digital
CeSC (Cambridge)
Software Sustainability Institute
How hard can a compute Grid be…? • Information Systems
– What resources are available
• Servers, CPUs, memory, storage, queues,
OS, applications, databases, …
• Monitoring and Discovery Systems
– What is the status of those resources • Queues empty/very full
• Machines running for a week
• Hard to tell for some applications…?
• Job scheduling/resource brokering – Please run these {jobs}
• Fastest, most secure/reliable, cheapest … – Jobs need inter-process communication…?
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com