Microsoft PowerPoint – COMP528 HAL09 MPI comms.pptx
Dr Michael K Bane, G14, Computer Science, University of Liverpool
m.k. .uk https://cgi.csc.liv.ac.uk/~mkbane/COMP528
COMP528: Multi-core and
Multi-Processor Programming
9 – HAL
https://www.mpi-forum.org/
Running MPI
• Implementation Dependent!
• Generally, you require
• An MPI implementation: see “module” for options
• Implementation = system integration + compiler wrapper
• Low level drivers for interconnect
• Wrapper to pick on specific headers and link to relevant lib
• To compile (with relevant module loaded)
• Gnu: mpicc
• Intel: mpiicc
• To launch (NB you should normally do this in batch)
• mpiexec -n 9 ./a.out (to run 9 processes)
• mpirun -np 8 ./a.out (alternative, for 8 processes)
• (occasionally something a lot more complicated!)
MPI messages
• Point to point communications
• One sender
• One receiver
• Pair-wise comms
• There are different flavours of “send” and of “receive”
• Collective communications
• All processes (within a communicator) participate
• Examples to follow
COMP328/COMP538 (c) mkbane, university of liverpool
MPI pt-2-pt: syntax
NAME
MPI_Send – Performs a blocking sending
SYNOPSIS
int MPI_Send(void *buf,
int count, MPI_Datatype datatype,
int dest,
int tag,
MPI_Comm comm)
INPUT PARAMETERS
buf – initial address of sending buffer
(choice)
count – number of elements in sending buffer
datatype
– datatype of each sending buffer element
(handle)
dest – rank of destination (integer)
tag – message tag (integer)
comm – communicator (handle)
NAME
MPI_Recv – Blocking receiving for a message
SYNOPSIS
int MPI_Recv(void *buf, int count, MPI_Datatype datatype,
int source,
int tag,
MPI_Comm comm,
MPI_Status *status)
OUTPUT PARAMETERS
buf – initial address of receiving buffer (choice)
status – status object (Status)
INPUT PARAMETERS
count – maximum number of elements in receiving buffer (integer)
datatype
– datatype of each receiving buffer element (handle)
source – rank of source (integer)
tag – message tag (integer)
comm – communicator (handle)
Message has “envelope”
• c.f. putting letter in general comms postbox
• How does it know where to go?
Put an address on it
• MPI is fussy
• A process will only receive wanted messages
by stating requirements of who sent it & what they
have sent
COMP328/COMP538 (c) mkbane, university of liverpool
MPI pt-2-pt: envelope
NAME
MPI_Send – Performs a blocking sending
SYNOPSIS
int MPI_Send(void *buf,
int count, MPI_Datatype datatype,
int dest,
int tag,
MPI_Comm comm)
INPUT PARAMETERS
buf – initial address of sending buffer (choice)
count – number of elements in sending buffer
datatype
– datatype of each sending buffer element (handle)
dest – rank of destination (integer)
tag – message tag (integer)
comm – communicator (handle)
NAME
MPI_Recv – Blocking receiving for a message
SYNOPSIS
int MPI_Recv(void *buf, int count, MPI_Datatype datatype,
int source,
int tag,
MPI_Comm comm,
MPI_Status *status)
OUTPUT PARAMETERS
buf – initial address of receiving buffer (choice)
status – status object (Status)
INPUT PARAMETERS
count – maximum number of elements in receiving buffer (integer)
datatype
– datatype of each receiving buffer element (handle)
source – rank of source (integer)
tag – message tag (integer)
comm – communicator (handle)
Messages are sent when
dest==source and tags & communicator match
We also like to ensure same data type
Wildflags are allowed – but totally discouraged except when totally necessary
MPI pt-2-pt: envelope
MESSAGE ENVELOPEMESSAGE DATA
NAME
MPI_Recv – Blocking receiving for a message
SYNOPSIS
int MPI_Recv(void *buf, int count, MPI_Datatype datatype,
int source,
int tag,
MPI_Comm comm,
MPI_Status *status)
OUTPUT PARAMETERS
buf – initial address of receiving buffer (choice)
status – status object (Status)
INPUT PARAMETERS
count – maximum number of elements in receiving buffer (integer)
datatype
– datatype of each receiving buffer element (handle)
source – rank of source (integer)
tag – message tag (integer)
comm – communicator (handle)
NAME
MPI_Send – Performs a blocking sending
SYNOPSIS
int MPI_Send(void *buf,
int count, MPI_Datatype datatype,
int dest,
int tag,
MPI_Comm comm)
INPUT PARAMETERS
buf – initial address of sending buffer (choice)
count – number of elements in sending buffer
datatype
– datatype of each sending buffer element (handle)
dest – rank of destination (integer)
tag – message tag (integer)
comm – communicator (handle)
COMP328/COMP538 (c) mkbane, university of liverpool
MPI example
#include
#include
#include
const int MAX_STRING = 100;
int main(void) {
char greeting[MAX_STRING];
int comm_sz;
int my_rank;
MPI_Init(NULL, NULL);
MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
if (my_rank != 0) {
sprintf(greeting, “Greetings from processor %s, process %d of %d!”, processor_name, my_rank, comm_sz);
MPI_Send(greeting, strlen(greeting)+1, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
} else {
printf(“Greetings from processor %s, process %d of %d!\n”, processor_name, my_rank, comm_sz);
for (int q = 1; q < comm_sz; q++) {
MPI_Recv(greeting, MAX_STRING, MPI_CHAR, q, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("%s\n", greeting);
}
}
MPI_Finalize();
return 0;
}
P Pacheco, "An Introduction to Parallel Programming",
Morgan Kaufmann, example 3.1
MPI example
#include
#include
#include
const int MAX_STRING = 100;
int main(void) {
char greeting[MAX_STRING];
int comm_sz;
int my_rank;
MPI_Init(NULL, NULL);
MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
if (my_rank != 0) {
sprintf(greeting, “Greetings from processor %s, process %d of %d!”, processor_name, my_rank, comm_sz);
MPI_Send(greeting, strlen(greeting)+1, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
} else {
printf(“Greetings from processor %s, process %d of %d!\n”, processor_name, my_rank, comm_sz);
for (int q = 1; q < comm_sz; q++) {
MPI_Recv(greeting, MAX_STRING, MPI_CHAR, q, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("%s\n", greeting);
}
}
MPI_Finalize();
return 0;
}
P Pacheco, "An Introduction to Parallel Programming",
Morgan Kaufmann, example 3.1
MPI example
#include
#include
#include
const int MAX_STRING = 100;
int main(void) {
char greeting[MAX_STRING];
int comm_sz;
int my_rank;
MPI_Init(NULL, NULL);
MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
if (my_rank != 0) {
sprintf(greeting, “Greetings from processor %s, process %d of %d!”, processor_name, my_rank, comm_sz);
MPI_Send(greeting, strlen(greeting)+1, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
} else {
printf(“Greetings from processor %s, process %d of %d!\n”, processor_name, my_rank, comm_sz);
for (int q = 1; q < comm_sz; q++) {
MPI_Recv(greeting, MAX_STRING, MPI_CHAR, q, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("%s\n", greeting);
}
}
MPI_Finalize();
return 0;
}
P Pacheco, "An Introduction to Parallel Programming",
Morgan Kaufmann, example 3.1
MPI example
#include
#include
#include
const int MAX_STRING = 100;
int main(void) {
char greeting[MAX_STRING];
int comm_sz;
int my_rank;
MPI_Init(NULL, NULL);
MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
if (my_rank != 0) {
sprintf(greeting, “Greetings from processor %s, process %d of %d!”, processor_name, my_rank, comm_sz);
MPI_Send(greeting, strlen(greeting)+1, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
} else {
printf(“Greetings from processor %s, process %d of %d!\n”, processor_name, my_rank, comm_sz);
for (int q = 1; q < comm_sz; q++) {
MPI_Recv(greeting, MAX_STRING, MPI_CHAR, q, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("%s\n", greeting);
}
}
MPI_Finalize();
return 0;
}
P Pacheco, "An Introduction to Parallel Programming",
Morgan Kaufmann, example 3.1
MPI example
#include
#include
#include
const int MAX_STRING = 100;
int main(void) {
char greeting[MAX_STRING];
int comm_sz;
int my_rank;
MPI_Init(NULL, NULL);
MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
if (my_rank != 0) {
sprintf(greeting, “Greetings from processor %s, process %d of %d!”, processor_name, my_rank, comm_sz);
MPI_Send(greeting, strlen(greeting)+1, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
} else {
printf(“Greetings from processor %s, process %d of %d!\n”, processor_name, my_rank, comm_sz);
for (int q = 1; q < comm_sz; q++) {
MPI_Recv(greeting, MAX_STRING, MPI_CHAR, q, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("%s\n", greeting);
}
}
MPI_Finalize();
return 0;
}
Functions to determine at
run-time some properties of the
given Communicator
MPI example
#include
#include
#include
const int MAX_STRING = 100;
int main(void) {
char greeting[MAX_STRING];
int comm_sz;
int my_rank;
MPI_Init(NULL, NULL);
MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
if (my_rank != 0) {
sprintf(greeting, “Greetings from processor %s, process %d of %d!”, processor_name, my_rank, comm_sz);
MPI_Send(greeting, strlen(greeting)+1, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
} else {
printf(“Greetings from processor %s, process %d of %d!\n”, processor_name, my_rank, comm_sz);
for (int q = 1; q < comm_sz; q++) {
MPI_Recv(greeting, MAX_STRING, MPI_CHAR, q, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("%s\n", greeting);
}
}
MPI_Finalize();
return 0;
}
Consider this running on 3 processes…
1. How many messages are being sent?
2. How many messages are being received?
MPI example
#include
#include
#include
const int MAX_STRING = 100;
int main(void) {
char greeting[MAX_STRING];
int comm_sz;
int my_rank;
MPI_Init(NULL, NULL);
MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
if (my_rank != 0) {
sprintf(greeting, “Greetings from processor %s, process %d of %d!”, processor_name, my_rank, comm_sz);
MPI_Send(greeting, strlen(greeting)+1, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
} else {
printf(“Greetings from processor %s, process %d of %d!\n”, processor_name, my_rank, comm_sz);
for (int q = 1; q < comm_sz; q++) {
MPI_Recv(greeting, MAX_STRING, MPI_CHAR, q, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("%s\n", greeting);
}
}
MPI_Finalize();
return 0;
}
Consider this running on 3 processes…
1. How many messages are being sent by each process?
Rank0?
Rank 1 or Rank 2?
2. How many messages are being received?
Rank1 or Rank2?
Rank0?
Recap
MPI_Send(&myID, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
Core 0 Core 1
0 1
MPI_Recv(&inputBuffer, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
0
1
Sending a message from one process to another:
POINT TO POINT COMMUNICATION
How do we find the syntax (& description) of the MPI_Send
and MPI_Recv functions?
Errors
• Generally MPI functions return an int
• C/C++:
int MPI_Recv(…)
• Good programming (& debugging) requires testing of return codes.
• Successful calls return value of MPI_SUCCESS
• Should test against this, taking appropriate action if rc != MPI_SUCCESS
• Some routines, upon error, provide return codes that give more info (see their
manual pages)
COMP328/COMP538 (c) mkbane, university of liverpool
Recap
MPI_Send(&myID, 10, MPI_INT, 1, 0, MPI_COMM_WORLD);
Core 0 Core 1
0 1
MPI_Recv(&inputBuffer, 10, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
0
1
Sending a message from one process to another:
POINT TO POINT COMMUNICATION
Start address
Messages only send one data-type
but can send several contiguous elements
e.g. a vector of integers OR of reals, but not a mix
Recap
MPI_Send(&myID, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
Core 0 Core 1
0 1
MPI_Recv(&inputBuffer, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
0
1
MPI_Send is sending to rank 1
MPI_Recv is receiving from rank 0
Both have tag=0
Both in same communicator
Thus message will be delivered (data from rank 0 to rank 1)
Tag?
• We will discuss ordering (lack of) etc re messages shortly
• You should be aware that process A may send lots of messages to process B
• Process B may be ready to accept several messages from process A
• And that we do not necessarily know (at any given point in the run) which is the
“next” message (e.g. consider process A does a different branch at an “if” stmt)
• In order to know which message is being sent (and thus how to receive it), we use
“tag” to differentiate them
• e.g. solving dynamic Ax=b for ‘x’ where either A or b might change in real time
• MPI_Recv(&buffer, maxN, MPI_FLOAT, processA, tag, MPI_COMM_WORLD, &stat)
• tag=100 might be for receiving new ‘A’
• tag=200 might be for receiving new ‘b’
COMP328/COMP538 (c) mkbane, university of liverpool
Summary
• Discussed 2 types of MPI comms patterns
• MPI point to point
• MPI collective communications
• Discussed pair-wise MPI point to point
• A sender process: MPI_Send
• A receiver process: MPI_Recv
• Message comprises
• Data
• Envelope – needs to match for message to “complete”
Questions via MS Teams / email
Dr Michael K Bane, Computer Science, University of Liverpool
m.k. .uk https://cgi.csc.liv.ac.uk/~mkbane