Slide 1
Non-Blocking
Communications
Reusing this material
This work is licensed under a Creative Commons Attribution-
NonCommercial-ShareAlike 4.0 International License.
http://creativecommons.org/licenses/by-nc-sa/4.0/deed.en_US
This means you are free to copy and redistribute the material and adapt and build on the
material under the following terms: You must give appropriate credit, provide a link to the
license and indicate if changes were made. If you adapt or build on the material you must
distribute your work under the same license as the original.
Acknowledge EPCC as follows: “© EPCC, The University of Edinburgh, www.epcc.ed.ac.uk”
Note that this presentation contains images owned by others. Please seek their permission
before reusing these images.
3
http://creativecommons.org/licenses/by-nc-sa/4.0/deed.en_US
http://creativecommons.org/licenses/by-nc-sa/4.0/deed.en_US
http://creativecommons.org/licenses/by-nc-sa/4.0/deed.en_US
http://creativecommons.org/licenses/by-nc-sa/4.0/deed.en_US
http://creativecommons.org/licenses/by-nc-sa/4.0/deed.en_US
http://creativecommons.org/licenses/by-nc-sa/4.0/deed.en_US
1
5
2
3
4
0 Communicator
Deadlock
4
Completion
• The mode of a communication determines when its
constituent operations complete.
– i.e. synchronous / asynchronous
• The form of an operation determines when the procedure
implementing that operation will return
– i.e. when control is returned to the user program
5
Blocking Operations
• Relate to when the operation has completed.
• Only return from the subroutine call when the operation
has completed.
• These are the routines you used thus far
– MPI_Ssend
– MPI_Recv
6
Non-Blocking Operations
• Return straight away and allow the sub-program to
continue to perform other work. At some later time the
sub-program can test or wait for the completion of the
non-blocking operation.
7
Beep!
Non-Blocking Operations
• All non-blocking operations should have matching wait
operations. Some systems cannot free resources until
wait has been called.
• A non-blocking operation immediately followed by a
matching wait is equivalent to a blocking operation.
• Non-blocking operations are not the same as sequential
subroutine calls as the operation continues after the call
has returned.
8
Non-Blocking Communications
• Separate communication into three phases:
• Initiate non-blocking communication.
• Do some work (perhaps involving other communications?)
• Wait for non-blocking communication to complete.
9
1
5
2
3
4
0 Communicator
Non-Blocking Send
10
1
5
2
3
4
0 Communicator
Non-Blocking Receive
11
Handles used for Non-blocking Comms
• datatype same as for blocking (MPI_Datatype or
INTEGER).
• communicator same as for blocking (MPI_Comm or
INTEGER).
• request MPI_Request or INTEGER.
• A request handle is allocated when a communication is
initiated.
12
Non-blocking Synchronous Send
• C:
int MPI_Issend(void* buf, int count,
MPI_Datatype datatype, int dest,
int tag, MPI_Comm comm,
MPI_Request *request)
int MPI_Wait(MPI_Request *request,
MPI_Status *status)
• Fortran:
MPI_ISSEND(buf, count, datatype, dest,
tag, comm, request, ierror)
MPI_WAIT(request, status, ierror)
13
Non-blocking Receive
• C:
int MPI_Irecv(void* buf, int count,
MPI_Datatype datatype, int src,
int tag, MPI_Comm comm,
MPI_Request *request)
int MPI_Wait(MPI_Request *request,
MPI_Status *status)
• Fortran:
MPI_IRECV(buf, count, datatype, src,
tag, comm, request, ierror)
MPI_WAIT(request, status, ierror)
14
Blocking and Non-Blocking
• Send and receive can be blocking or non-blocking.
• A blocking send can be used with a non-blocking receive,
and vice-versa.
• Non-blocking sends can use any mode – synchronous,
buffered or standard
• Synchronous mode affects completion, not initiation.
15
Communication Modes
NON-BLOCKING OPERATION MPI CALL
Standard send MPI_ISEND
Synchronous send MPI_ISSEND
Buffered send MPI_IBSEND
Receive MPI_IRECV
16
Completion
• Waiting versus Testing.
• C:
int MPI_Wait(MPI_Request *request,
MPI_Status *status)
int MPI_Test(MPI_Request *request,
int *flag,
MPI_Status *status)
• Fortran:
MPI_WAIT(handle, status, ierror)
MPI_TEST(handle, flag, status, ierror)
17
MPI_Request request;
MPI_Status status;
if (rank == 0)
{
MPI_Issend(sendarray, 10, MPI_INT, 1, tag,
MPI_COMM_WORLD, &request);
Do_something_else_while Issend_happens();
// now wait for send to complete
MPI_Wait(&request, &status);
}
else if (rank == 1)
{
MPI_Irecv(recvarray, 10, MPI_INT, 0, tag,
MPI_COMM_WORLD, &request);
Do_something_else_while Irecv_happens();
// now wait for receive to complete;
MPI_Wait(&request, &status);
}
18
Example (C)
integer request
integer, dimension(MPI_STATUS_SIZE) :: status
if (rank == 0) then
CALL MPI_ISSEND(sendarray, 10, MPI_INTEGER, 1, tag,
MPI_COMM_WORLD, request, ierror)
CALL Do_something_else_while Issend_happens()
! now wait for send to complete
CALL MPI_Wait(request, status, ierror)
else if (rank == 1) then
CALL MPI_IRECV(recvarray, 10, MPI_INTEGER, 0, tag,
MPI_COMM_WORLD, request, ierror)
CALL Do_something_else_while Irecv_happens()
! now wait for receive to complete
CALL MPI_Wait(request, status, ierror)
endif
19
Example (Fortran)
Multiple Communications
• Test or wait for completion of one message.
• Test or wait for completion of all messages.
• Test or wait for completion of as many messages as
possible.
20
in
in
in
Process
Testing Multiple Non-Blocking Comms
21
Combined Send and Receive
• Specify all send / receive arguments in one call
– MPI implementation avoids deadlock
– useful in simple pairwise communications patterns, but not as generally
applicable as non-blocking
int MPI_Sendrecv(void *sendbuf, int sendcount, MPI_Datatype sendtype,
int dest, int sendtag,
void *recvbuf, int recvcount, MPI_Datatype recvtype,
int source, int recvtag,
MPI_Comm comm, MPI_Status *status);
MPI_SENDRECV(sendbuf, sendcount, sendtype, dest, sendtag,
recvbuf, recvcount, recvtype, source, recvtag,
comm, status, ierror)
22
Exercise
Rotating information around a ring
• See Exercise 4 on the sheet
• Arrange processes to communicate round a ring.
• Each process stores a copy of its rank in an integer
variable.
• Each process communicates this value to its right
neighbour, and receives a value from its left neighbour.
• Each process computes the sum of all the values
received.
• Repeat for the number of processes involved and print out
the sum stored at each process.
23
Possible solutions
• Non-blocking send to forward neighbour
– blocking receive from backward neighbour
– wait for forward send to complete
• Non-blocking receive from backward neighbour
– blocking send to forward neighbour
– wait for backward receive to complete
• Non-blocking send to forward neighbour
• Non-blocking receive from backward neighbour
– wait for forward send to complete
– wait for backward receive to complete
24
Notes
• Your neighbours do not change
– send to left, receive from right, send to left, receive from right, …
• You do not alter the data you receive
– receive it
– add it to you running total
– pass the data unchanged along the ring
• You must not access send or receive buffers until
communications are complete
– cannot read from a receive buffer until after a wait on irecv
– cannot overwrite a send buffer until after a wait on issend
25