DSCC 201/401
Tools and Infrastructure for Data Science
February 17, 2021
ssh web browser
BlueHive Resource Allocation
bluehive
terminal session on bluehive
default
FastX server
interactive interactive
FastX node (1 core, 2GB)
Slurm Server
compute node
(with requested resources)
Linux File System Hierarchy Tree Structure
3
Navigating the Tree – Linux File Commands
• Listing files: ls
• Help on a command: man
• Print working directory: pwd
• Change directory: cd
• Remove a file: rm
• Copy a file: cp
• Make a directory: mkdir
• Remove a directory: rmdir
• Shortcuts: .., ., ~, and –
• BlueHive: /home, /scratch, and /public
4
Working with Files
• Wildcards: *, ?
• Viewing files: cat, more, less
• Beginning and end of files: head, tail • Counting: wc
• Searching for text and files: grep, find • Pipes and directors: |, >
• Sorting: sort
• Useful commands: date, who, history • File editors: nano, vi, vim, and emacs • Searching: find
5
Linux Process Control and Limits
• What is currently running?: ps
• What is running in the background?: jobs • Interrupting a command: ^C
• Current processes running on system: top • Current disk space and limits: quota
• Current space used in directory: du
• Current memory usage: free
• Information on CPUs: cat /proc/cpuinfo
• Standard output redirection: >
• Standard error redirection: 2>
6
Remotely Copying Files
• Secure copy local file to remote with scp
scp localfile netid@bluehive.circ.rochester.edu:
• Secure copy remote file to local file with scp
scp netid@bluehive.circ.rochester.edu:remotefile .
7
Bash
• Bash is a shell (most commonly the default shell in Linux)
• Supports execution of commands interactively (as we have seen) and execution of commands from a file (known as shell scripts)
User Space (Bash is here)
Linux Kernel
RAM
CPU
Storage
Server
Network
8
Environment Variables
• Common environment variables
• $HOME – Home directory
• $USER – User name
• $PATH – Which paths will contain the executables • $TERM – Terminal type
• $SHELL – Shell in use • Useful commands
• echo • env
9
Bash Scripting on the Command Line Interface
• Working with multiple files
for file in *.c; do mv $file c_source; done
10
Bash Scripting on the Command Line Interface
• Now as a Bash script
#!/bin/bash
mkdir S_source
for file in *.S; do mv $file S_source; done
11
Bash Scripting on the Command Line Interface
• Improved format
#!/bin/bash
echo “creating directory”
mkdir S_source
echo “moving files”
for file in *.S
do mv $file S_source
done
12