MPI and Types
aka “What if everything isn’t together?” https://warwick.ac.uk/fac/sci/dcs/teaching/material/cs402/ 01/02/2022 ● CS402/922 High Performance Computing ● ●
01/02/2022
Copyright By PowCoder代写 加微信 powcoder
Recap on Message Passing Interface (MPI)
In case it got lost in the post!
• MPI sends data from one processor to anotherà MPI_Send and MPI_Recv
• Base version is blocking (function waits for a response)
• Required info includes:
• Number of elements of data
• The type of data to be sent
• Source/Destination processor
• Communicator (the group of processors that could be accessed) • Tag (to distinguish between different messages)
• A status message (for MPI_Recv only)
01/02/2022
Different modes for MPI_Send and MPI_Recv Plenty of options to choose from…
• Standard version are blocking functions
• We can have non-blocking versions
• The way in which data is sent and processed can be altered:
• Buffered
• Synchronous • Ready
(Standard)
Synchronous
Blocking (Send)
Blocking (Recv)
Non-blocking (Send)
MPI_IBsend
MPI_ISsend
MPI_IRsend
Non-blocking (Recv)
MPI_IBrecv
MPI_ISrecv
MPI_IRrecv
01/02/2022
⌨ … Whoops, wrong typing…
• MPI messages need to be common between different
processors
• They may require different sizes, so need a unified interface
• Some are only relevant to:
• FORTRANàMPI_COMPLEX,MPI_LOGICAL…
• CàMPI_C_BOOL,MPI_INT64_T…
• C++àMPI_CXX_BOOL,MPI_CXX_DOUBLE_COMPLEX…
• Many more that can be used, make sure to use the right type!
MPI Spec (Data TypesàPage 34)
https://www.mpi-forum.org/docs/mpi-4.0/mpi40-report.pdf
01/02/2022
What MPI_Send is actually doing…
I write, I compile, I run, that’s it!
• Say we wanted to send 4 floats from processor 0 to 1
float * data = {1.0f, 2.0f, 3.0f, 4.0f}; MPI_Send(data, 4, MPI_FLOAT, 1, 0, MPI_COMM_WORLD);
(MPI Envelope)
01/02/2022
What MPI_Send is actually doing…
I write, I compile, I run, that’s it!
• Say we wanted to send 4 floats from processor 0 to 1
float * data = {1.0f, 2.0f, 3.0f, 4.0f}; MPI_Send(data, 4, MPI_FLOAT, 1, 0, MPI_COMM_WORLD);
(MPI Envelope)
01/02/2022
What MPI_Send is actually doing…
I write, I compile, I run, that’s it!
• Say we wanted to send 4 floats from processor 0 to 1
float * data = {1.0f, 2.0f, 3.0f, 4.0f}; MPI_Send(data, 4, MPI_FLOAT, 1, 0, MPI_COMM_WORLD);
(MPI Envelope)
01/02/2022
Non-contiguous memory
In case it got lost in the post!
• Data shown in the previous example is con$guous • All the data is uniform with no gaps
• SomeOmes our data is not in conOguous memory • (An array of) struct or custom classes
• Block of memory manually addressed
• If we ran the same thing, you would get garbled data
16 bits per float 4 bits per char
01/02/2022
Dealing with Non-contiguous memory
Not all data is the same size!
• 3 different options to deal with this:
1. Call a MPI_Send for each data block occupying contiguous blocks • Lots of messages means lots of congestion
• Lots of messages means lots of confusion if not managed well
• Lots of messages means lots of overhead for sending multiple messages
2. Copy all the data into a contiguous block, and send that
• Better than the first option as one message is being sent
• Lots of time needs to be spent copying memory
• Not great if dealing with large amounts of data, where memory is limited
3. Derived datatypes…
01/02/2022
Derived Datatypes
That’s the thing from the previous slide; he said the thing!
• We can generate our own datatypes
• Useful as we can send a single message with different types of data
and/or non-contiguous memory
• This can improve performance (but is not guaranteed)
• Derived types need to be committed (using MPI_Type_commit)
• 5 key derived types
MPI_Type_contiguous MPI_Type_vector MPI_Type_create_indexed_block MPI_Type_indexed MPI_Type_create_struct
More complex
01/02/2022
MPI_Type_contiguous
A bit like MPI_Type++!
• Useful to be specify a collection of elements with a variable amount of counts
• Allows for better reuse and less calculations
int MPI_Type_contiguous(int count, MPI_Datatype oldtype, MPI_Datatype *newtype)
Returns 0 if success, otherwise error number
Number of The old data type The new data type blocks
(not negative)
01/02/2022
MPI_Type_contiguous
A bit like MPI_Type++!
int MPI_Type_contiguous(int c3ount, MPI_DaMtPaIt_yFpLeOAoTldtype, MPI&_fDlaotaattCyopneti*gnueowutsype)
Returns 0 if success, otherwise error number
Number of The old data type The new data type blocks
(not negative)
01/02/2022
MPI_Type_contiguous
A bit like MPI_Type++!
MPI_Datatype floatContiguous;
MPI_Type_contiguous(3, MPI_FLOAT, &floatContiguous); MPI_Type_commit(&floatContiguous);
/* floatContiguous =
{(MPI_FLOAT, 0), (MPI_FLOAT, 16), (MPI_FLOAT, 32)} */
MPI_Type_free(&floatContiguous);
01/02/2022
MPI_Type_vector
A vector called Victor!
• Consecutive blocks of contiguous data
• Each block between contiguous blocks need to be equally spaced,
the same size, and all the same type
int MPI_Type_vector(int count, int blocklength, int stride, MPI_Datatype oldtype, MPI_Datatype *newtype)
Number of blocks (not negative)
Number of elements in each block
(not negative)
Number The old data type The new data type of
elements between blocks
01/02/2022 Slide 15
MPI_Type_vector
A vector called Victor!
int MPI_Type_vector(int 3count, int blo2cklength, int s4tride, MPI_DaMtPaIt_yFpLeOAoTldtype, MPI_D&aftlaotaytpVeec*tnoerwtype)
Number of blocks (not negative)
Number of elements in each block
(not negative)
Number of elements between blocks
The old data type
The new data type
blocklength
01/02/2022
MPI_Type_vector
A vector called Victor!
MPI_Datatype floatVector;
MPI_Type_vector(3, 2, 4, MPI_FLOAT, &floatVector); MPI_Type_commit(&floatVector);
/* floatVector =
{(MPI_FLOAT, 0), (MPI_FLOAT, 16),
(MPI_FLOAT, 64), (MPI_FLOAT, 80),
(MPI_FLOAT, 128), (MPI_FLOAT, 144)} */
MPI_Type_free(&floatVector);
01/02/2022
MPI_Type_create_indexed_blockA block called Blocky!
• Sometimes the data doesn’t start at the beginning, or might have
multiple offsets
• Requires a the same number of elements in each block, and them all to be the same type
int MPI_Type_create_indexed_block(int count, int blocklength, const int offsets[], MPI_Datatype oldtype, MPI_Datatype *newtype);
The new data type
Number of blocks (not negative, size of offsets)
Number of elements in each block
(not negative)
Array of offsets, in mulXples of oldtype
The old data type
01/02/2022 Slide 18
MPI_Type_create_indexed_block
A block called Blocky!
int MPI_Type_create_indexed_block(int c3ount, int bloc2klength, const{1i,nt5,of1f0s}ets[], MPI_DaMtPaIt_yFpLeOAoTldtype, MPI_Datatype *newtype);
&floatCreateIndexBlock
The new data type
Number of blocks (not negative, size of offsets)
Number of elements in each block
(not negative)
Array of offsets, in multiples of oldtype
The old data type
blocklength
01/02/2022
MPI_Type_create_indexed_blockA block called Blocky!
MPI_Datatype floatCreateIndexBlock;
MPI_Type_create_indexed_block(3, 2, {1, 5, 10}, MPI_FLOAT, &floatCreateIndexBlock); MPI_Type_commit(&floatCreateIndexBlock);
/* floatCreateIndexBlock =
{(MPI_FLOAT, 16), (MPI_FLOAT, 32),
(MPI_FLOAT, 80), (MPI_FLOAT, 96),
(MPI_FLOAT, 160), (MPI_FLOAT, 176)} */
MPI_Type_free(&floatCreateIndexBlock);
01/02/2022 Slide 20
MPI_Type_indexed
An index called Indexy?
• Sometimes the data blocks aren’t the same size…
• Requires all to be the elements to be the same type
int MPI_Type_indexed(int count, const int blocklengths[], const int offsets[], MPI_Datatype oldtype, MPI_Datatype *newtype);
The new data type
Number of blocks (not negative, size of blocklengths and offsets)
Array of block lengths (all elements cannot be negative)
Array of offsets, in mulXples of oldtype
The old data type
01/02/2022 Slide 21
MPI_Type_indexed
An index called Indexy?
3 {3, 1, 2} {1, 6, 10} MPI_FLOAT
int MPI_Type_indexed(int count, const int blocklengths[], const int offsets[], MPI_Datatype oldtype,
MPI_Datatype *newtype); &floatIndexed
Number of blocks (not negative, size of blocklengths and offsets)
Array of block lengths (all elements cannot be negative)
Array of offsets, in multiples of oldtype
The old data type
The new data type
blocklength[0]
blocklength[1] blocklength[2]
01/02/2022
MPI_Type_indexed
An index called Indexy?
MPI_Datatype floatIndexed;
MPI_Type_indexed(3, {3, 1, 2}, {1, 6, 10}, MPI_FLOAT, &floatIndexed); MPI_Type_commit(&floatIndexed);
/* floatIndexed =
{(MPI_FLOAT, 16), (MPI_FLOAT, 32), (MPI_FLOAT, 48), (MPI_FLOAT, 96),
(MPI_FLOAT, 160), (MPI_FLOAT, 176)} */
MPI_Type_free(&floatIndexed);
01/02/2022
MPI_Type_create_struct
An struct called Structy?
• Sometimes even the blocks are different types
• Requires the elements within each individual block need to be
the same type
int MPI_Type_create_struct(int count, const int blocklengths[], const int offsets[], const MPI_Datatype oldtypes[], MPI_Datatype *newtype);
The new data type
Number of blocks (not negative, size of blocklengths , offsets and oldtypes)
Array of block lengths (all elements cannot be negative)
Array of offsets,
Array of data types for each block
oldtypes[]
01/02/2022 Slide 24
MPI_Type_create_struct
An struct called Structy?
int MPI_Type_create_struct(int 3count, const in{t3,bl1o,ck2l}engths[], cons{t4,in1t8,of3f1s}ets[], c{oMnPsIt_FMLPOIA_TD,atMaPtIy_pCeHAR, , MPI_Datatype *newtype);
The new data type
Number of blocks (not negaXve, size of blocklengths , offsets and oldtypes)
Array of block lengths (all elements cannot be negative)
Array of offsets,
Array of data types for each block
oldTypes[0]=MPI_FLOAT oldTypes[1]=MPI_CHAR oldTypes[2]=MPI_INT
&structType
blocklength[0]
blocklength[1] blocklength[2]
01/02/2022
MPI_Type_create_struct
An struct called Structy?
MPI_Datatype structType;
MPI_Type_create_struct (3, {3, 1, 2}, {4, 18, 31}, {MPI_FLOAT, MPI_CHAR, MPI_INT}, &structType); MPI_Type_commit(&structType);
/* structType =
{(MPI_FLOAT, 16), (MPI_FLOAT, 32), (MPI_FLOAT, 48), (MPI_CHAR, 66),
(MPI_INT, 83), (MPI_INT, 91)} */
MPI_Type_free(&structType);
01/02/2022 Slide 26
Overview of Complex Derived Datatypes
That… that was too much info!
Same type in each block
Same number of elements in each block
Same distance between consecutive block
MPI_Type_vector
MPI_Type_create_indexed_block
MPI_Type_indexed
MPI_Type_create_struct
01/02/2022
Interesting related reads
Some of this might even be fun…
• Zero (cost) copy with MPI Derived Datatypes
• T. Hoefler and S. Gottlieb. Parallel Zero-Copy Algorithms for Fast Fourier Transform and Conjugate Gradient Using MPI Datatypes. In
Recent Advances in the Message Passing Interface, pages 132—141, 2010, Springer, Berlin, Heidelberg • MPI Derived Datatypes performance improvements
• W. Gropp, E. Lusk and D. Swider. Improving the Performance of MPI Derived Datatypes. In Proceedings of the Third MPI Developer’s and User’s Conference, pages 25—30, March 1999, Springer, Berlin, Heidelberg
• J. L. Träff, R. Hempel, H. Ritzdorf and F. Zimmermann. Flattening on the Fly: Efficient Handelling of MPI Derived Datatypes. In European Parallel Virtual Machine / Message Passing Interface Users’ Group Meeting (EuroPVM/MPI 1999): Recent Advances in Parallel Virtual Machine and Message Passing Interface, pages 109—116, September 1999, Springer, Berlin, Heidelberg
Next lecture: Advanced MPI
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com