Introduction to Message Passing Interfaces
aka “Dear other processors, I hope this data finds you well…”
https://warwick.ac.uk/fac/sci/dcs/teaching/material/cs402/ 24/01/2022 ● CS402/922 High Performance Computing ● ●
Copyright By PowCoder代写 加微信 powcoder
25/01/2022
Parallelism on Supercomputers
What’s wrong with OpenMP?
• OpenMP parallelises on shared memory (Cache and RAM)
• Great on a single computer
• Not so useful on supercomputersàdata may be stored on another computer
• OpenMP implemented at compile-time • Useful as it makes code easier
• Very little fine-grained control
• What if I could send some data for another processor to act on later?
25/01/2022
Message Passing Interface
I don’t need stamps do I?
• We can pass “messages” between processors
• Because messages are being passed, processors don’t need to be on the same computer
• Sent over network if a remote processor • Stored in memory if a local processor
• Most commonly usedàMessage Passing Interface (MPI)
25/01/2022
What is MPI?
• MPI is an interface!
Isn’t “Message Passing Interface” enough‽
• Different (and wildly varying) implementations exist
• Most common examples include OpenMPI and Intel MPI (IMPI)
• Most common version of the interfaceàMPI 3.0 • MPI 4.0 released in July 2021
• Interfaces have been built for FORTRAN, C and C++
https://www.mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf
25/01/2022
Getting started with MPI
First, lets set up the environment…
• MPI requires specialised compilers
• To load the compilers on the DCS system module load cs402-mpi
• Also need to include the header file #include
FORTRAN à mpifort
C à mpicc
C++ à mpicxx
25/01/2022
MPI Initialisation
• 3 key elements required to an MPI program
• We can initialise the MPI environment
• Gain some basic info about the parallel environment
• At the end, we need to stop the MPI environment
Then we set up the WORLD!!!
#include
int main(int argc, char** argv) {
int world_size, world_rank; //Initialise the MPI environment/world
MPI_Init(NULL, NULL);
//Get the number of processors
MPI_Comm_size(MPI_COMM_WORLD, &world_size); //Get the processor number/rank
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
/* Do some computation here */
//Finish the MPI environemnt
MPI_Finalize();
25/01/2022
MPI_COMM_WORLD
Wait, did the previous slide say “set up the world”‽
MPI_COMM_WORLD
• MPI allows you to split the processors up into different sectionsàMPI Communicators (MPI_Comm)
• MPI_COMM_WORLD is created by MPI_Init • Represents all processors
• Static value
• You can create your own communicators if you want à MPI_Comm_split
• Unless you are doing something complex, stick with MPI_COMM_WORLD
25/01/2022
Building MPI programs
• Added in a print statement • To compile:
Brick by brick, line by line…
mpicc –o testMPI testMPI.c
mpirun –n 4 ./testMPI
Specify you want to run the code in an MPI environment
Specify the MPI
processors file
executable
#include
#include
int main(int argc, char** argv) {
int world_size, world_rank;
//Initialise the MPI environment/world
MPI_Init(NULL, NULL);
//Get the number of processors
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
//Get the processor number/rank
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
//Print out some info
printf(“I am processor %d of %d\n”,
world_rank, world_size);
/* Do some computation here */
//Finish the MPI environemnt
MPI_Finalize();
25/01/2022
Building MPI programs
Brick by brick, line by line…
mpirun –n 4 ./testMPI
• This program will output the following:
I am processor 2 of 4
I am processor 1 of 4
I am processor 3 of 4
I am processor 4 of 4
#include
#include
int main(int argc, char** argv) {
int world_size, world_rank;
//Initialise the MPI environment/world
MPI_Init(NULL, NULL);
//Get the number of processors
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
//Get the processor number/rank
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
//Print out some info
printf(“I am processor %d of %d\n”,
world_rank, world_size);
/* Do some computation here */
//Finish the MPI environemnt
MPI_Finalize();
25/01/2022
Passing data between processors
Lets pass some messages… I get the name now!
• Rememberàprocessors could be anywhere on our network
• Send packets of data across the network from one processor to another
• Some info is required in order to make sure the message is sent properly
Message Length Flags & Tags Rank Context etc…
Payload (Data)
25/01/2022
Sending Messages
I got gel pens, and glitter, and stickers, and coloured paper, and…
• We can send data using the (conveniently named) function MPI_Send
• This packages the data in the “envelope”, and gives it to the communications device (network card)
• The communications device manages the routing (so you don’t have to!)
• The receiving processor will send an acknowledgement
25/01/2022
void* data,
int count,
MPI_Datatype datatype,
int destination,
MPI_Comm communicator)
So who is this for?
A pointer to the data being sent.
The number of data elements to be sent.
The datatype of the data to be sent.
25/01/2022
void* data,
int count,
MPI_Datatype datatype,
int destination,
MPI_Comm communicator)
SIDETRACK!!!
MPI_SHORT à short int MPI_INT à int
MPI_LONG à long int MPI_UNSIGNED à unsigned int MPI_FLOAT à float MPI_DOUBLE à double MPI_BYTE à char
MPI Spec (Data TypesàPage 34)
https://www.mpi-forum.org/docs/mpi-4.0/mpi40-report.pdf
25/01/2022
void* data,
int count,
MPI_Datatype datatype,
int destination,
MPI_Comm communicator)
So who is this for?
A pointer to the data being sent.
The number of data elements to be sent.
The datatype of the data to be sent. The processor ID to send the data.
An identifier tag.
The communicator to be used (usually MPI_COMM_WORLD).
25/01/2022
Receiving Messages
What a colourful letter, why is it covered in glitter‽
• A process needs to know when it should expect data
• We can receive data using the (equally conveniently
named) function MPI_Recv
• The receiving processor will send an
acknowledgement, containing a status marker
• MPI_Recv looks like MPI_Send…
25/01/2022
void* data,
int count,
MPI_Datatype datatype,
int source,
MPI_Comm communicator
MPI_Status* status)
A pointer to where the data is being stored.
The number of data elements to be received.
The datatype of the data to be received.
The processor ID where the data is being received from.
An identifier tag.
The communicator to be used
(usually MPI_COMM_WORLD). The status to be sent back.
25/01/2022 Slide 17
MPI_Send and MPI_Recv
MPI_Send( MPI_Recv(
Here you go!
void* data,
int count,
MPI_Datatype datatype,
void* data,
int count,
MPI_Datatype datatype,
int destination, int source,
int tag, int tag,
MPI_Comm communicator) MPI_Comm communicator
MPI_Status* status)
25/01/2022
Example MPI program
/* Initial setup */
Put it together and what do you get!
/* Compute – Everyone send the square of there rank to processor 0 */
if(world_rank == 0) {
int i, data;
for (i=1; i < world_size; i++) {
MPI_Recv(&data, 1, MPI_INT, i, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Processor 0 received the value %d from processor %d\n", data, i); }
int data = world_rank * world_rank;
MPI_Send(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD); printf("Processor %d sent %d to processor 0\n", world_rank, data);
/* Further compute and finalise */
25/01/2022 Slide 19
Example MPI program
/* Initial setup */
Put it together and what do you get!
/* Compute - Everyone send the square of there rank to processor 0 */
if(world_rank == 0) {
int i, data;
for (i=1; i < world_size; i++) {
MPI_Recv(&data, 1, MPI_INT, i, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Processor 0 received the value %d from processor %d\n", data, i); }
int data = world_rank * world_rank;
MPI_Send(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD); printf("Processor %d sent %d to processor 0\n", world_rank, data);
/* Further compute and finalise */
25/01/2022 Slide 20
Example MPI program
/* Initial setup */
Put it together and what do you get!
/* Compute - Everyone send the square of there rank to processor 0 */
if(world_rank == 0) {
int i, data;
for (i=1; i < world_size; i++) {
MPI_Recv(&data, 1, MPI_INT, i, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Processor 0 received the value %d from processor %d\n", data, i); }
int data = world_rank * world_rank;
MPI_Send(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD); printf("Processor %d sent %d to processor 0\n", world_rank, data);
/* Further compute and finalise */
25/01/2022 Slide 21
Example MPI program
/* Initial setup */
Put it together and what do you get!
/* Compute - Everyone send the square of there rank to processor 0 */
if(world_rank == 0) {
int i, data;
for (i=1; i < world_size; i++) {
MPI_Recv(&data, 1, MPI_INT, i, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Processor 0 received the value %d from processor %d\n", data, i); }
int data = world_rank * world_rank;
MPI_Send(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD); printf("Processor %d sent %d to processor 0\n", world_rank, data);
/* Further compute and finalise */
25/01/2022 Slide 22
Example MPI program
/* Initial setup */
Put it together and what do you get!
/* Compute - Everyone send the square of there rank to processor 0 */
if(world_rank == 0) {
int i, data;
for (i=1; i < world_size; i++) {
MPI_Recv(&data, 1, MPI_INT, i, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Processor 0 received the value %d from processor %d\n", data, i); }
int data = world_rank * world_rank;
MPI_Send(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD); printf("Processor %d sent %d to processor 0\n", world_rank, data);
/* Further compute and finalise */
25/01/2022 Slide 23
Example MPI program
/* Initial setup */
Put it together and what do you get!
/* Compute - Everyone send the square of there rank to processor 0 */
if(world_rank == 0) {
int i, data;
for (i=1; i < world_size; i++) {
MPI_Recv(&data, 1, MPI_INT, i, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Processor 0 received the value %d from processor %d\n", data, i); }
int data = world_rank * world_rank;
MPI_Send(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD); printf("Processor %d sent %d to processor 0\n", world_rank, data);
/* Further compute and finalise */
25/01/2022 Slide 24
Example MPI program
/* Initial setup */
Put it together and what do you get!
/* Compute - Everyone send the square of there rank to processor 0 */
if(world_rank == 0) {
int i, data;
for (i=1; i < world_size; i++) {
MPI_Recv(&data, 1, MPI_INT, i, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Processor 0 received the value %d from processor %d\n", data, i); }
int data = world_rank * world_rank;
MPI_Send(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD); printf("Processor %d sent %d to processor 0\n", world_rank, data);
/* Further compute and finalise */
25/01/2022 Slide 25
Example MPI program
/* Initial setup */
Put it together and what do you get!
data da1ta 4 9
/* Compute - Everyone send the square of there rank to processor 0 */
if(world_rank == 0) {
int i, data;
for (i=1; i < world_size; i++) {
MPI_Recv(&data, 1, MPI_INT, i, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Processor 0 received the value %d from processor %d\n", data, i); }
int data = world_rank * world_rank;
MPI_Send(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD); printf("Processor %d sent %d to processor 0\n", world_rank, data);
/* Further compute and finalise */
25/01/2022 Slide 26
Example MPI program
/* Initial setup */
Put it together and what do you get!
data data 4 9
/* Compute - Everyone send the square of there rank to processor 0 */
if(world_rank == 0) {
int i, data;
for (i=1; i < world_size; i++) {
MPI_Recv(&data, 1, MPI_INT, i, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Processor 0 received the value %d from processor %d\n", data, i); }
int data = world_rank * world_rank;
MPI_Send(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD); printf("Processor %d sent %d to processor 0\n", world_rank, data);
/* Further compute and finalise */
25/01/2022 Slide 27
Example MPI program
/* Initial setup */
Put it together and what do you get!
/* Compute - Everyone send the square of there rank to processor 0 */
if(world_rank == 0) {
int i, data;
for (i=1; i < world_size; i++) {
MPI_Recv(&data, 1, MPI_INT, i, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Processor 0 received the value %d from processor %d\n", data, i); }
int data = world_rank * world_rank;
MPI_Send(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD); printf("Processor %d sent %d to processor 0\n", world_rank, data);
/* Further compute and finalise */
25/01/2022 Slide 28
Example MPI program
/* Initial setup */
Put it together and what do you get!
/* Compute - Everyone send the square of there rank to processor 0 */
if(world_rank == 0) {
int i, data;
for (i=1; i < world_size; i++) {
MPI_Recv(&data, 1, MPI_INT, i, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Processor 0 received the value %d from processor %d\n", data, i); }
int data = world_rank * world_rank;
MPI_Send(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD); printf("Processor %d sent %d to processor 0\n", world_rank, data);
/* Further compute and finalise */
25/01/2022 Slide 29
Example MPI program
Put it together and what do you get!
I am processor 2 of 4
Processor 2 sent 4 to processor 0
I am processor 0 of 4
I am processor 1 of 4
Processor 1 sent 1 to processor 0
I am processor 3 of 4
Processor 3 sent 9 to processor 0
Processor 0 received the value 1 from processor 1
Processor 0 received the value 4 from processor 2
Processor 0 received the value 9 from processor 3
25/01/2022
Issues with MPI_Send and MPI_Recv
Why is this always near the end of the lecture...
• Both MPI_Send and MPI_Recv are blocking functions
• The process will wait until the data (and envelope) are stored, or an
acknowledge has been received
• We can have non-blocking versions of both
• MPI_Isend(const void *buf, int count, MPI_Datatype
datatype, int dest, int tag, MPI_Comm comm,
MPI_Request *request)
• MPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Request *request)
Used to query whether the request has been completed (using MPI_Wait and MPI_Test)
25/01/2022
Communication Modes
I thought we just send it‽
• MPI has flexibility in how it the communications are achieved
• BufferedàSend instructions can start at any point and may complete
before a corresponding receive instruction is executed
• SynchronousàSend instructions can start at any point, but can only complete successfully when a receive instruction is executed
• ReadyàSend can only start if the receive has been executed
• Can be incorporated to blocking or non-blocking
• Prepend the letter of the mode to the instruction • ExamplesàMPI_IBsend,MPI_Rsend,MPI_Ssend
25/01/2022
Interesting related reads
Some of this might even be fun...
• MPI Tutorial websiteàhttps://mpitutorial.com/ • Exascale MPI Workshop (ExaMPI)à
https://sc21.supercomputing.org/session/?sess=sess430
Next lecture: MPI and Types
25/01/2022
Student Feedback
Make your voice heard!
Teaching Methods Feedback Questionnaire
Moodle à CS402 à
CS402 Week 3 Teaching Methods Feedback
Closes: 31st January 2022
https://moodle.warwick.ac.uk/mod/questionnaire/view.php?id=1566049
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com