MSc in High Performance Computing
MSc in High Performance Computing
Notes for first practical to set up machine accounts
Session 1: Informatics DICE desktop accounts
An Informatics DICE account is required to log in to desktop machines in the Informatics
training rooms.
You can set your initial DICE password at https://pp.inf.ed.ac.uk/.
If you do not have access to a DICE desktop, you can test your account by logging in
to student.ssh.inf.ed.ac.uk
This is only a gateway system with minimal software – for serious work you should
then log in to student.compute.ed.ac.uk
For documentation, see http://computing.help.inf.ed.ac.uk/.
Session 2: Applying for a Cirrus account
Cirrus is a national tier-2 HPC machine, hosted and managed by EPCC, which we will be
using throughout the MSc. You need to apply for an account on Cirrus and this is a two-step
process:
1. Sign up for a SAFE account at https://www.archer.ac.uk/tier2
Email address: MUST be your sms email e.g. s1234567@sms.ed.ac.uk
Nationality: MUST be your actual nationality – don’t just use default (United
Kingdom)
Institution: University of Edinburgh
Department: EPCC
Phone number/address: UK address/number
Opt out of user Emails: Don’t check this box
2. Once you have received your SAFE password by email you need to request a
machine account
Log into SAFE (https://www.archer.ac.uk/tier2)
Under the “Login accounts” drop down, click “Request login account”
Select the project d167 from the drop down list.
Machine name is Cirrus
Your username MUST be your UUN e.g. s1234567
You will then receive an email informing you that your Cirrus account has been created and
details about how to log in to the machine.
Session 2: Applying for a ARCHER account
Although you should be able to do all your semester-1 practical work on Cirrus, we encourage
you to obtain an account on ARCHER as well.
To apply for an account on the UK national supercomputer, ARCHER, is very similar to
Cirrus, again this is a two-step process (note the different URLs):
3. Sign up for a SAFE account at https://www.archer.ac.uk/safe/signup.jsp
Email address: MUST be your sms email e.g. s1234567@sms.ed.ac.uk
Nationality: MUST be your actual nationality – don’t just use the default (UK)
Institute for RAE reporting: University of Edinburgh (scroll down drop-down
menu)
Institution: University of Edinburgh
Department: EPCC
Phone number/address: UK address/number
Opt out of user Emails: Don’t check this box
https://pp.inf.ed.ac.uk/
https://www.archer.ac.uk/tier2
mailto:s1234567@sms.ed.ac.uk
https://www.archer.ac.uk/tier2
https://www.archer.ac.uk/safe/signup.jsp
mailto:s1234567@sms.ed.ac.uk
4. Once you have received your SAFE password by email you need to request a
machine account
Log into SAFE (https://www.archer.ac.uk/safe/login.jsp)
On the Main page, click the “Request New Account” button
Select the project d167 from the drop down list.
Machine name is archer
Your username must be your UUN e.g. s1234567
You will later receive an email informing you when your ARCHER account has been created
which will include details of how to log on to the machine. Whilst Cirrus accounts are created
almost instantaneously, ARCHER accounts can take up to a few hours to create.
More details at http://www.archer.ac.uk/documentation/safe-guide/safe-guide-users.php
Detailed ARCHER documentation at http://www.archer.ac.uk/documentation/user-guide
Session 3: Logging into Cirrus and running a job
Logging in to Cirrus
Once you have received the email with your new account details then follow the instructions
to retrieve your password.
You will need to use the Secure Shell, ssh, to connect to Cirrus.
Linux: open a terminal and use the command line.
Mac: as for Linux, but you also have to install “Xquartz” from
https://www.xquartz.org/.
Windows: install MobaXterm from https://mobaxterm.mobatek.net/.
Next, type: ssh –XY cirrus-msc.epcc.ed.ac.uk (the –XY flags forward any UNIX X
window graphics to your local machine.) You can change your password with the passwd
command
If you are very unfamiliar with UNIX then it is worth jumping to section 5 and working through
the shell and editors documents before coming back to this section.
Open a terminal window and start a text editor e.g. emacs:
[cirrus-login2]$ emacs &
Emacs is a very sophisticated edirto but can be quite daunting for new users – you may find
gedit easier to use:
[cirrus-login2]$ gedit &
Write a simple program (e.g. one that just prints “hello world”) in C, C++ or Fortran.
There are links Unix and editors guides on the Learn MSc Programme Information hub, see
section 6 of this worksheet for more information.
Preparation to compile your code
Cirrus uses the module environment for selecting the appropriate compiler and libraries. We
need to load the mpt and intel-compilers modules in order to have available the SGI
parallel libraries and Intel compiler.
[cirrus-login2]$ module load mpt
[cirrus-login2]$ module load intel-compilers-17
To automate this, these two commands can be added into your .bash_profile
https://www.archer.ac.uk/safe/login.jsp
http://www.archer.ac.uk/documentation/safe-guide/safe-guide-users.php
http://www.archer.ac.uk/documentation/user-guide
https://www.xquartz.org/
https://mobaxterm.mobatek.net/
Compiling your code
To compile your code
Fortran: use mpif90
o mpif90 -o hello hello.f90
C: use mpicc
o mpicc -cc=icc -o hello hello.c
C++: use mpicxx
o mpicxx -cc=icpc -o hello hello.cpp
Both GNU and Intel compilers are available, we will use the Intel compilers here and in future
practicals you will explore switching between compilers and the differences that this results in.
Running on the login node
Now you have compiled your code you can run it on the login node like any other program:
[cirrus-login2]$ ./hello
If you have followed the instructions above then you have compiled your code to use MPI, this
is one of the key parallel libraries which you will be learning about in great detail during the
MSc. We will now run four copies of your program in parallel on the login node:
[cirrus-login2]$ mpirun –np 4 ./hello
Running on the backend compute nodes
To run your executable on the backend compute nodes, you need to package it up into a
submission script and submit this to the queue. There is an example you can copy and use:
[cirrus-login2]$ cp /lustre/home/shared/subhello.pbs subhello.pbs
(Note this script assumes your executable is called hello, if it is something different then you
will need to edit the script.)
A queuing system called PBS is used, you can submit a job to it via the qsub command
[cirrus-login2]$ qsub -q R348954 subhello.pbs
You can use qstat –u $USER to list your queued or running jobs, or qstat to list all jobs.
Important: Once this has run you will find the output in a file named helloworld.oxxx (where
xxx is a unique number). There will also be a helloworld.exxx containing any error messages.
Important: We are submitting with -q R348954 which tells the queue system to use pre-
reserved compute nodes for your job. This is known as a reservation and we will often use
this during practicals in order to avoid having to wait in the machine’s main queue. The exact
reservation code will change from practical to practical.
You can always submit a job to the general Cirrus queue using:
[cirrus-login2]$ qsub subhello.pbs
Without a reservation, your job will be in the same queue as all other users and could take a
long time to run depending on the overall usage of Cirrus.
Charging code
All jobs on Cirrus (and on ARCHER) are charged to an account or budget of CPU time. The
default is the general d167 budget, specified by the following line in subhello.pbs:
#PBS -A d167
All students have access to their own budget indicated by your UUN, e.g. d167-s1234567.
Edit the script to charge to your own personal budget and check that you can still run your
program using qsub. When running jobs on the MSc you should always use your own budget.
Note that, if we provide you with a script, it will probably be set up to charge to the default
d167 budget so you will need to edit the PBS scripts before running.
Session 5: Using ARCHER
The general principals are the same as Cirrus but with some minor differences. There are
different compiler commands such as cc for C, CC for C++ and ftn for Fortran. After
creating the executable hello, write a script called hello.pbs containing the following:
#!/bin/bash –login
cd $PBS_O_WORKDIR
./hello
You can submit this to the backend via
-bash-4.1$ qsub -Vl select=1,walltime=0:01:0 -A d167 hello.pbs
On ARCHER you may also use qstat –u $USER to list your jobs, or qstat to list all jobs.
Section 6: Shell and editors guide
There are guides for working with the UNIX shell and common editors on the MSc
Programmes Information Hub Learn course. These can be found under the induction page
(Introduction to machine accounts pane) and are prerequisite information for many of the
labs. It is therefore worth working through this material to ensure you are fully familiar with it.
The shell PDF is an introduction to the Unix Shell
The introduction to Unix editors PDF provides an overview of the most common ways
of editing files on Unix.
Section 7: Other Useful information
Register for EASE
http://www.ed.ac.uk/schools-departments/information-services/services/help-
consultancy/help-services/online-help-guidance/students/it-help/guides/ease-guide
Register for Wireless
http://www.ed.ac.uk/schools-departments/information-
services/computing/connecting/registering
http://www.ed.ac.uk/schools-departments/information-services/services/help-consultancy/help-services/online-help-guidance/students/it-help/guides/ease-guide
http://www.ed.ac.uk/schools-departments/information-services/services/help-consultancy/help-services/online-help-guidance/students/it-help/guides/ease-guide
http://www.ed.ac.uk/schools-departments/information-services/computing/connecting/registering
http://www.ed.ac.uk/schools-departments/information-services/computing/connecting/registering