Cloud Computing & Big Data, Fall 2020
Columbia University – Graduate School of Engineering and Applied Science
Cloud Computing Special Topics
Andrei Papancea ’15
Columbia University, MS Computer Science
NLX Inc, Co-Founder & CEO
HW2 Questions?
1. Platform-independent Microservices
2. DevOps
3. Infrastructure as Code
4. Data Science as Code
Microservices using Docker
● What is Docker?
○ Software platform that allows you to build, test, and
deploy applications quickly
○ The code, OS, libraries, etc. are bundled together as a
unit called container, which is independently deployed
at scale
○ Think of it as Open Source Lambda functions
Docker Containers
Why Docker?
● All the benefits of microservices PLUS ● Platform-independent
○ You can port your code between Cloud providers or accounts with less effort than porting provider-specific services
● Widespread developer support
Docker on AWS
● The painful way
○ Run and manage your own Docker “swarm” on EC2
■ A cluster of Docker images orchestrated to run across one or more machines
● The easy way
○ Use Elastic Container Service (ECS), a
highly-scalable, managed Docker container service from AWS
ECS Architecture
ECS Example
DEMO
Recap:
1. You can use Docker to build your own microservices, which you can port across platforms.
2. You can use ECS to run your Docker containers at scale.
1. Platform-independent Microservices
2. DevOps
3. Infrastructure as Code
4. Data Science as Code
What is DevOps?
“DevOps is a set of practices that automates the processes between software development and IT teams, in order that they can build, test, and release software faster and more reliably.” — Atlassian
Why do you need it?
● All the benefits of microservices PLUS ● Platform-independent
○ You can port your code between Cloud providers or accounts with less effort than porting provider-specific services
● Widespread developer support
Recap:
1. DevOps is a set of practices to automate how you build, test, and release software.
2. On AWS you can use CodeBuild and CodePipeline to automate your code builds and deployments.
1. Platform-independent Microservices 2. DevOps
3. Infrastructure as Code
4. Data Science as Code
✓ DevOps helps with automating code deployments.
What about infrastructure?
Serverless Website Infrastructure
S3
API Gateway
Lambda Lambda Lambda Lambda
As your infrastructure gets more complex, the need for automating its deployment increases.
AWS CloudFormation
● “Infrastructure as Code”
● Infrastructure, configuration, and security all encoded in
config files called “templates”
● Reproducible infrastructure safely and reliably
AWS CloudFormation
AWS CloudFormation
CloudFormation Template
AWS CloudFormation
DEMO
Recap:
1. Automating infrastructure is a great way to reproduce infrastructure safely and reliably
2. You can use CloudFormation on AWS to automate your entire infrastructure.
1. Platform-independent Microservices 2. DevOps
3. Infrastructure as Code
4. Data Science as Code
Thank you! QUESTIONS?
Sources:
1. https://aws.amazon.com/codebuild/
2. https://aws.amazon.com/codepipeline/
3. https://aws.amazon.com/cloudformation/
4. https://www.slideshare.net/AmazonWebServices/build-cicd-pipeline-for-containe
r-presentation-slides
5. https://www.slideshare.net/AmazonWebServices/introducing-amazon-sagemak
er
6. https://aws-de-media.s3.amazonaws.com/images/AWS_Summit_Berlin_2016/s
essions/pushing_the_boundaries_1300_microservices_on_aws.pdf
7. https://github.com/ndrppnc/serverless-cicd
8. https://github.com/alik0211/pokedex