PowerPoint Presentation
PUBLIC
++
Real World HPC Security
Warwick University
1st February 2018
John Fitzpatrick
PUBLIC
Once upon a time…
PUBLIC
1. Introduction
2. HPC Overview
3. Authentication
4. Privilege Escalation
5. Outside of the HPC world
6. Wrap up
Agenda:
PUBLIC
1. Introduction
2. HPC Overview
3. Authentication
4. Privilege Escalation
5. Outside of the HPC world
6. Wrap up
Agenda:
PUBLIC
Cray | 299,008 CPU cores | 40PB 1.4 TB/s IO Lustre
710 TB Memory (32GB+6GB/node) | 20+ petaFLOPS | 18,688 nodes
PUBLIC
++
HPC Usage
+ Weather Forecasting
+ Data Mining
+ Cryptanalysis
+ Nuclear Weapons Simulation
+ Molecular Dynamics
+ Oil & Gas
PUBLIC
++
Security
PUBLIC
++
HPC Architecture
PUBLIC
1. Introduction
2. HPC Overview
3. Authentication
4. Privilege Escalation
5. Outside of the HPC world
6. Wrap up
Agenda:
PUBLIC
++
Workload/Resource Managers
PUBLIC
++
Workload/Resource Managers
PUBLIC
++
Workload/Resource Managers
PUBLIC
++
Workload/Resource Managers
PUBLIC
++
@Warwick
Slurm is an open source, fault-tolerant and
highly scalable cluster management and job
scheduling system for large and small Linux
clusters
Moab HPC Suite is a workload and resource
orchestration platform that automates the scheduling,
managing, monitoring, and reporting of HPC
workloads on massive scale
PUBLIC
++
SLURM
PUBLIC
++
SLURM
Commands
PUBLIC
++
Example SLURM Message
E…K.@.@..%………|…0V-.<0. ................................ ............auth/munge.........M UNGE:AwQDAAANogYonuFTIPGguqSU7b2 DxkdB/yJNwMbTSxdU0sx1tAkU9cWL7RP f+jX3PhdCLLNz3yMIRzC9Q+zNdaa+ie6 carmfu5bw4PqWQKE3gkMVDZtOrBI=:.. .....................id.....slur m1.............................. ................................ ................./usr/bin/id.... ............./home/user1........ ................................ ................................ ..?............................. ........................f.. PUBLIC ++ Example SLURM Message E...K.@.@..%.........|...0V-.<0. ................................ ............auth/munge.........M UNGE:AwQDAAANogYonuFTIPGguqSU7b2 DxkdB/yJNwMbTSxdU0sx1tAkU9cWL7RP f+jX3PhdCLLNz3yMIRzC9Q+zNdaa+ie6 carmfu5bw4PqWQKE3gkMVDZtOrBI=:.. .....................id.....slur m1.............................. ................................ ................./usr/bin/id.... ............./home/user1........ ................................ ................................ ..?............................. ........................f.. PUBLIC ++ Example SLURM Message E...K.@.@..%.........|...0V-.<0. ................................ ............auth/munge.........M UNGE:AwQDAAANogYonuFTIPGguqSU7b2 DxkdB/yJNwMbTSxdU0sx1tAkU9cWL7RP f+jX3PhdCLLNz3yMIRzC9Q+zNdaa+ie6 carmfu5bw4PqWQKE3gkMVDZtOrBI=:.. .....................id.....slur m1.............................. ................................ ................./usr/bin/id.... ............./home/user1........ ................................ ................................ ..?............................. ........................f.. PUBLIC ++ Example SLURM Message E...K.@.@..%.........|...0V-.<0. ................................ ............auth/munge.........M UNGE:AwQDAAANogYonuFTIPGguqSU7b2 DxkdB/yJNwMbTSxdU0sx1tAkU9cWL7RP f+jX3PhdCLLNz3yMIRzC9Q+zNdaa+ie6 carmfu5bw4PqWQKE3gkMVDZtOrBI=:.. .....................id.....slur m1.............................. ................................ ................./usr/bin/id.... ............./home/user1........ ................................ ................................ ..?............................. ........................f.. PUBLIC ++ Munge in action user1@slurm1:/tmp> munge -s “Warwick MUNGE example”
MUNGE:AwQDAAAdrmatMHFDGbhF/agNUUcbTCfaoJLP4J8D0GkIMY3NZPA+7wCPN8ijmaQJRWt5rkMsXVmKc
E9RVbOQ7d3DY2BHK/58QV2cqcuzv6Zxo9pFJl6ZpnlRCsiUhrTS4NZZDMkQIyXd:
PUBLIC
++
Unmunge in action
user1@slurm1:/tmp> echo “MUNGE:AwQDAAAdrmatMHFDGbhF/agNUUcbTCfaoJLP4J8D0GkI
MY3NZPA+7wCPN8ijmaQJRWt5rkMsXVmKcE9RVbOQ7d3DY2BHK/58QV2cqcuzv6Z
xo9pFJl6ZpnlRCsiUhrTS4NZZDMkQIyXd:” | unmunge
STATUS: Success (0)
ENCODE_HOST: slurm1 (10.178.175.17)
ENCODE_TIME: 2018-01-31 12:08:51 (1517400531)
DECODE_TIME: 2018-01-31 12:10:08 (1517400608)
TTL: 300
CIPHER: aes128 (4)
MAC: sha1 (3)
ZIP: none (0)
UID: user1 (1001)
GID: users (100)
LENGTH: 22
Warwick MUNGE example
PUBLIC
++
Munge info
user1@slurm1:/tmp> ls -la /usr/local/var/run/munge/
total 12
drwxr-xr-x 2 root root 4096 Jan 28 12:23 .
drwxr-xr-x 3 root root 4096 Jul 24 2013 ..
-rw-r–r– 1 root root 5 Jan 28 12:23 munged.pid
srwxrwxrwx 1 root root 0 Jan 28 12:23 munge.socket.2
slurm1:/usr/local/etc/munge # ls -la
total 12
drwx—— 2 root root 4096 Jul 24 2013 .
drwxr-xr-x 5 root root 4096 Jan 28 11:40 ..
-rw——- 1 root root 1024 Jul 24 2013 munge.key
PUBLIC
++
Moab
PUBLIC
++
Moab::mauth
[user1@moab ~]$ ls -la /opt/moab/bin/mauth
-rwsr-x–x. 1 root root 130384507 Sep 18 2014 /opt/moab/bin/mauth
Mauth (for Moab)
PUBLIC
++
Moab::mauth
[user1@moab ~]$ ls -la /opt/moab/bin/mauth
-rwsr-x–x. 1 root root 130384507 Sep 18 2014 /opt/moab/bin/mauth
Mauth (for Moab)
[user1@moab ~]$ ls -la /opt/moab/etc/.moab.key
-r——–. 1 root root 31 Sep 17 2014 /opt/moab/etc/.moab.key
PUBLIC
++
Moab::mauth
PUBLIC
++
Moab::mauth
PUBLIC
++
Cray::aprun
PUBLIC
++
Cray::aprun
Run as UID=0
PUBLIC
++
Cray::aprun
PUBLIC
++
Cray::aprun
Run as UID=0
PUBLIC
++
TRQAUTHD (TORQUE)
PUBLIC
++
Trqauthd (TORQUE)
PUBLIC
NeedProper validation of the messages – don’t trust user supplied
input
Generate and use your own keys, and keep them secret
PUBLIC
1. Introduction
2. HPC Overview
3. Authentication
4. Privilege Escalation
5. Outside of the HPC world
6. Wrap up
Agenda:
PUBLIC
++
Embedded devices
PUBLIC
++
System Imaging
PUBLIC
++
DDN
+ DataDirect Networks (DDN) – Storage
PUBLIC
++
DDN :: Default Credentials
root:$1$Euo5wva3$OHbI5ew.Vojh**********:16526:0:99999:7:::
ddn:$1$hRQTHVz9$ExF9hMUxn6gk**********:16526:0:99999:7:::
user:$1$5RiEj1yl$J0hiuuncUJHm**********:16526:0:99999:7:::
firmware:$1$cenUmzbv$nFMqerCXlV9X**********:16526:0:99999:7:::
diag:$1$5RiEj1yl$J0hiuuncUJHm**********:16526:0:99999:7:::
stats:$1$x9dzJ6UA$uI7upgmkJ7yp**********:16526:0:99999:7:::
PUBLIC
++
DDN :: Default Credentials
/home$ cat user/.ssh/id_rsa
—–BEGIN RSA PRIVATE KEY—–
MIIEpgIBAAKCAQEAyoSW9x6DucKz3W/1TyX+EPUcwIAOh6cFvsy6n1qIYYDiXtBf
buOk/a8i3ZZJtGNhxeKJCk5+Wk9HQOwQz3lWNKKmq+waYDBuVaUK1QZeVLNLRAyF
…
home$ cat stats/.ssh/id_rsa
—–BEGIN RSA PRIVATE KEY—–
MIIEpgIBAAKCAQEAyoSW9x6DucKz3W/1TyX+EPUcwIAOh6cFvsy6n1qIYYDiXtBf
buOk/a8i3ZZJtGNhxeKJCk5+Wk9HQOwQz3lWNKKmq+waYDBuVaUK1QZeVLNLRAyF
…
home$ cat diag/.ssh/id_rsa
—–BEGIN RSA PRIVATE KEY—–
MIIEpAIBAAKCAQEAtU3CCh287eMt6temAT3IzMr3JlwFEzvLfq915rEtzdGiJh6Q
kVGZNHIlx3+X3dxEFCfD2XzitBEtkUZ8y1y43p7dtXNwJqKt7VEpuuosEZp5yQyk
…
$ cat ddn/.ssh/id_rsa
—–BEGIN RSA PRIVATE KEY—–
MIIEowIBAAKCAQEA3dwed/Xw59DkKdfo1TGCY+yDXkujWxG0xNcn+UBN4aG7wGzk
0tcNLUbN/PpKEltUCxK/dBb9AZ/wD2OPyFxzfpHUFV5OCXP3V0uQx/0kahEnL0Ud
…
PUBLIC
++
DDN :: Insecure Firmware Upload Mechanism
ddn> up con local file myfirmware.tgz
janus_update.sh
/bin/bash
exit(1)
PUBLIC
++
GPFS / Spectrum Scale
+ General Parallel File System / Spectrum Scale
+ Parallel file system developed by IBM
PUBLIC
++
GPFS / Spectrum Scale
GPFS Client
GPFS Utilities
mmchfileset
mmcrsnapshot
mmdelsnapshot
mmdf
mmedquota
mmgetacl
mmlsdisk
mmlsfileset
mmlsfs
mmlsmgr
mmlspolicy
mmlspool
mmlsquota
mmlssnapshot
mmputacl
Mmsnapdir
…
..
.
PUBLIC
++
GPFS / Spectrum Scale
GPFS Client
GPFS Utilities
mmchfileset
mmcrsnapshot
mmdelsnapshot
mmdf
mmedquota
mmgetacl
mmlsdisk
mmlsfileset
mmlsfs
mmlsmgr
mmlspolicy
mmlspool
mmlsquota
mmlssnapshot
mmputacl
Mmsnapdir
…
..
.
PUBLIC
++
GPFS / Spectrum Scale
$ mmlscluster
PUBLIC
++
GPFS / Spectrum Scale
$ mmlscluster “;PUT COMMAND HERE#”
PUBLIC
Don’t trust third party components
Root anywhere probably means root everywhere
PUBLIC
1. Introduction
2. HPC Overview
3. Authentication
4. Privilege Escalation
5. Outside of the HPC world
6. Wrap up
Agenda:
PUBLIC
++
Outside of the HPC World
+ Valid approach to most technology
PUBLIC
++
Other MWR Research
PUBLIC
++
Other MWR Research
PUBLIC
John.Fitzpatrick@mwrinfosecurity.com
@j0hn__f
www.mwrinfosecurity.com / @mwrinfosecurity
labs.mwrinfosecurity.com / @mwrlabs
Questions?