DSCC 201/401
Tools and Infrastructure for Data Science
February 22, 2021
Bash
• Bash is a shell (most commonly the default shell in Linux)
• Supports execution of commands interactively (as we have seen) and execution of commands from a file (known as shell scripts)
User Space (Bash is here)
Linux Kernel
RAM
CPU
Storage
Server
Network
2
Environment Variables
• Common environment variables
• $HOME – Home directory
• $USER – User name
• $PATH – Which paths will contain the executables • $TERM – Terminal type
• $SHELL – Shell in use • Useful commands
• echo • env
3
Bash Scripting on the Command Line Interface
• Working with multiple files
for file in *.c; do mv $file c_source; done
4
Bash Scripting on the Command Line Interface
• Now as a Bash script
#!/bin/bash
mkdir S_source
for file in *.S; do mv $file S_source; done
5
Bash Scripting on the Command Line Interface
• Improved format
#!/bin/bash
echo “creating directory”
mkdir S_source
echo “moving files”
for file in *.S
do
mv $file S_source
done
6
Bash Shell Scripting
• Create a variable: name=variable (no white space allowed!) • No data types for variables
• Variables are referenced by $ in front of variable name
• Variables cannot begin with numbers
#!/bin/bash
statement=”Hello, world!”
echo $statement
pi=3.14159
echo $pi
7
Bash Shell Scripting
• Consequences of no typing
• Doing “calculations” with Bash (only practical for integers)
#!/bin/bash #!/bin/bash #!/bin/bash
pi=3.1459 pi=3.1459 pi=3 two_pi=2*$pi two_pi=$((2*pi)) two_pi=$((2*pi)) echo $two_pi echo $two_pi echo $two_pi
8
• Command substitution: $( ) • Arithmetic expansion: $(( )) • Parameter expansion: ${ }
cs=$(ls .)
echo $cs
a=3
b=4
ae=$((a+b))
echo $ae
name=my_file.txt
cp $name ${name}.copy
base=${name%.*}
cp $base $base.copy
Bash Shell Scripting
9
Conditionals
• Conditionals: if-then-else
• Can also use elif for multiple conditionals
#!/bin/bash
if [ $# != 0 ]; then
if [ $1 = hi ]; then
echo ‘first argument was “hi”‘
else
echo “first argument was $1”
fi
else
echo “no arguments given”
fi
10
Conditionals
• Test for existence of file and directory
#!/bin/bash
if [ -f $file ]
then
echo “$file exists!”
else
echo “$file does not exist!”
fi
if [ -d $dir ]
then
echo “$dir exists!”
else
echo “$dir does not exist!”
fi
11
Loops
• Loops can provide for repeated statements • For loop format: for, do, done
#!/bin/bash
for file in $(ls .)
do
cp $file ${file}.copy
done
12
Loops
• Loops can provide for repeated statements
• While loop format: while, do, done
• Comparisons: less than (lt), less than or equal to (lt), greater than (gt), greater than or equal to (ge)
i=1
while [ $i -le 10 ]
do
echo $i
i=$((i+1))
done
13
Slurm Commands
• sinfo – status of partitions
• squeue – list jobs in queue
• sbatch – submit job to cluster
• scancel – cancel jobs running or in queue
14
Slurm Scripts
#!/bin/bash
#SBATCH -p debug
#SBATCH -c 1
#SBATCH –mem=2GB
#SBATCH -t 1:00:00
#SBATCH -J test
#SBATCH -e test.err
#SBATCH -o test.out
#
echo “start at $(date)”
hostname
echo “end at $(date)”
15
More Information
• Linux Tutorial:
• http://www.ee.surrey.ac.uk/Teaching/Unix/
• Shell programming with Bash:
• http://matt.might.net/articles/bash-by-example/
16