MPI tutorial
CMPSC 450
Message Passing Interface
• Specification for developers and users
• Open-source and commercial MPI implementations • Libraries
• OpenMPI, MPICH, MVAPICH, Intel MPI, Cray MPI, etc. • History
• Programming model
• Predominantly distributed memory, no sharing
• Slides based on LLNL MPI tutorial
• https://computing.llnl.gov/tutorials/mpi/
CMPSC 450
MPI components
• Environment management routines
• Point-to-point communication routines
• Collective communication
• Group and communicator management, virtual topologies
• Syntax differs based on programming language, search online to look it up
CMPSC 450
MPI program structure
SPMD
Single Program Multiple Data
CMPSC 450
MPI Hello world in C
CMPSC 450
Point-to-point communication routines
• Process src sends some data to process dest • Process dest receives data from process src
Process 1
Send(data)
Process 8
Recv(data)
• Identifying tasks/processes?
• How to describe data?
• How will receiver organize messages? • When is an operation complete?
CMPSC 450
Data types
• In MPI call, data is specified by [address, count, type]
• Data type can be
• Standard scalar types: MPI_INT, MPI_DOUBLE, MPI_CHAR, … • Array of data types
• Strided block of data types
• Arbitrary structure of data types
CMPSC 450
Message tags and status objects
• Message tags
• Every message has a user-defined integer ID • Wildcard: MPI_ANY_TAG
• Status objects
• Opaque structures for querying error and other conditions
CMPSC 450
Buffering
• System buffer holds data in transit
• System buffer space, parameters depend on MPI
implementation
• Application buffer is data managed by programmer (user)
CMPSC 450
MPI blocking send
MPI_Send(buffer, count, type, dest-rank, tag, comm)
Process 1:
MPI_Send(A, n, MPI_INT, 10, tag, comm)
Process 10:
MPI_Recv(A, n, MPI_INT, 1, tag, comm, &stat)
• A blocking send only “returns” after it is safe to modify application buffer. Does not imply message has been received.
CMPSC 450
MPI blocking receive
MPI_Recv(buffer, count, type, src-rank, tag, comm, status)
Process 1:
MPI_Send(A, n, MPI_INT, 10, tag, comm)
Process 10:
MPI_Recv(A, n, MPI_INT, 1, tag, comm, &stat)
• A blocking receive “returns” after matching message has been received. Can query status for more information.
CMPSC 450
Non-blocking operations
• Non-blocking ops return immediately with handles • Wait on handles
• May poll (“test”) instead of waiting
• May poll or wait on multiple requests
MPI_Request req;
MPI_Status stat;
MPI_Isend(buf, n, MPI_INT, dest, tag, comm, &req);
// do not use buf, but can do some other computation
MPI_Wait(&req, &stat);
Allows computation-communication overlap
CMPSC 450
Other communication modes
• Synchronous sends (MPI_Ssend): send completes before receive begins
• Buffered mode (MPI_Bsend): User-supplied buffer
• Ready mode (MPI_Rsend): User guarantees matching receive has
posted
• Non-blocking versions of above
• MPI_Recv accepts messages sent in any mode
• MPI_Sendrecv: Simultaneous send and receive
CMPSC 450
Order, Fairness
• Order
• MPI guarantees messages will not overtake each other
• Does not apply if there are multiple threads participating in communication
• Fairness
• MPI does not guarantee fairness, up to programmer to prevent starvation
CMPSC 450
Beware of deadlock
• Unsafe send/receive orderings • How to avoid?
Process 1:
Recv (data 2) Send (data 2)
Process 2:
Recv (data 1) Send (data 1)
Process 1:
Send (data 2) Recv (data 2)
Process 2:
Send (data 1) Recv (data 1)
CMPSC 450
Avoiding deadlock
• Use safe orderings
• Use MPI_Sendrecv
Process 1:
Send (data 2) Recv (data 2)
Process 2:
Recv (data 1) Send (data 1)
CMPSC 450
Collective operations
CMPSC 450
Collectives
• Synchronization: barrier
• Data movement: broadcast, scatter/gather, all-to-all
• Collective computation: reductions
• Collective operations are blocking
• Must involve all processors in the scope of communicator
• Do not require tags as arguments
• Optimized by MPI library developers (using strategies such as recursive doubling, binary tree-structured communication, pipelining)
• Use collectives as much as possible, avoid use of sends and recvs
CMPSC 450
Example programs
• Hello world
• Send & receive messages
CMPSC 450