计算机代写 P Y THON PAR ALLEL COMPU TING

P Y THON PAR ALLEL COMPU TING
Lecture 10

SERIAL COMPUTING

• 1kpiecespuzzle • Takes 10 hours

• Orangeandgreensharethepuzzleonthesametable
• Takes 6 hours
(not 5 due to communication & contention)
PARALLELISM ON SHARED MEMORY

• Lackofseats(Resourcelimit)
• Morecontentionamongpeople
THE MORE, THE BE T TER??

• Scalableseats(ScalableResource)
• Lesscontentionfromprivatememoryspaces
PARALLELISM ON DISTRIBUTED SYSTEMS

HOW TO SHARE THE PUZZLE?

PYTHON PARALLEL COMPUTING
• Two basic approaches
• Shared memory
• Distributed memory

• Provides shared memory physically or virtually
• Pros – Easy to use
• Cons – Limited Scalability, High coherence overhead
DSM (DISTRIBUTED SHARED MEMORY)

SHARED MEMORY

DIS TRIBU TED MEMORY

• Pros–Scalable,Flexible
• Cons–Someonesaysit’smoredifficultthanDSM
MESSAGE PASSING

How many people are working? What is my role?
How to send and receive data?
WHAT WE NEED TO KNOW…

PARALLELIZATION AND AMDAHL’S LAW
• Want to leverage parallelization as much as possible
• Often we cannot obtain perfect (linear) speedups, e.g., communication or
global logic
• Amdahl’s law is a simple law to get an idea of the speedup:
• N: number of processors
• P: fraction of program that can be parallelized

PARALLELIZATION AND AMDAHL’S LAW

MPI (MESSAGE PASSING IN TERFACE)
• Widely used standard
• Forprogrammingdistributed-memory,multipleinstruction–multiple data (MIMD) systems

POINT TO POINT COMMUNICATION
• Processesshouldcoordinatetheiractivitiesbyexplicitlysendingandreceiving messages

MPI (MESSAGE PASSING IN TERFACE)
• MPIoperatesasfollows:
• Process A decides a message needs to be sent to process B.
• Process A packs up all of its necessary data into a buffer for process B.
• Process A indicates that the data should be sent to process B by calling the Send function.
• Process B needs to acknowledge it wants to receive the message by calling the Recv function.

MPI (MESSAGE PASSING IN TERFACE)
• Everytimeaprocesssendsamessage,theremustbeaprocessthatalso indicates it wants to receive the message.
• Therefore, calls to Send and Recv are always paired.

MPI (MESSAGE PASSING IN TERFACE)
• The number of processes Is fixed when an MPI program is started
• Eachoftheprocessesisassignedauniqueintegerstartingfrom0.
• This integer is know as the rank of the process and is how each process is identified when sending and receiving messages
• MPIprocessesarearrangedinlogicalcollectionsknown as communicators.
• There is one special communicator (MPI.COMM_WORLD) that exists when an MPI program starts, which contains all the processes in the MPI program.

• MPI – Message Passing Interface
• Library standard defined by a committee of vendors, implementers, and parallel
programmers
• Used to create parallel programs based on message passing
• Portable: one standard, many implementations
• Available on almost all parallel machines in C and Fortran
• De facto standard platform for the HPC community

MPI (MESSAGE PASSING IN TERFACE)
• MPIprovidesafewmethodsonacommunicator:
• Get_size() – returns the total number of processes contained in the communicator
(the size of the communicator).
• Get_rank() – returns the rank of the calling process within the communicator, between 0 and (size-1)
• Send() – sends content to a process
• Recv() – receives content from a process

• A synchronous communication is not complete until the message has been received
• An asynchronous communication completes before the message is received
S YNCHRONOUS VS. AS YNCHRONOUS MESSAGE PASSING

COMMUNIC ATION MODES
• Synchronous: completes once ack is received by sender • Asynchronous: 3 modes
• Standard send: completes once the message has been sent, which may or may not imply that the message has arrived at its destination
• Buffered send: completes immediately, if receiver not ready, MPI buffers the message locally
• Ready send: completes immediately, if the receiver is ready for the message it will get it, otherwise the message is dropped silently

BLOCKING VS. NON-BLOCKING
• Blocking,meanstheprogramwillnotcontinueuntilthecommunicationis completed
• Synchronous communication
• Barriers: wait for every process in the group to reach a point in execution
• Non-Blocking,meanstheprogramwillcontinue,withoutwaitingforthe communication to be completed

MPI (MESSAGE PASSING IN TERFACE)

MPI FROM COMMAND LINE

ONE MPI PROGRAM, MULTIPLE MPI PROCESSES
• Makingeachprocesstoperformadifferentcomputation

ONE MPI PROGRAM, MULTIPLE MPI PROCESSES

POIN T- TO-POIN T COMMUNICATION
• Messagepassinginvolvestwoprocesses: • a sender (Send)
• a receiver (Recv)

POIN T- TO-POIN T COMMUNICATION

MPI. AN Y_SOURCE
• The receiving process does not always need to specify the source when issuing a Recv.
• Instead,theprocesscanacceptanymessagethatisbeingsentbyanother process.
• This is done by setting the source to MPI.ANY_SOURCE.

MPI. AN Y_SOURCE

• Sendmanydifferenttypesofmessagestoanotherprocess
• Insteadofhavingtogothroughextrameasurestodifferentiateallthese messages, MPI allows senders and receivers to also specify message IDs (known as tags) with the message.
• The receiving process can then request a message with a certain tag number and messages with different tags will be buffered until the process requests them.

• The status can provide useful information

NON-BLOCKING COMMUNICATION
• Inthepreviousexamples,thesenderandreceiverarenotabletoperformany action when sending or receiving a message.
• This can waste computation time while waiting for the call to complete.
• Non-blockingcommuncationavoidsthisissuebyusing
the Isend and Irecv methods, which start to send and receive operations and then return immediately to continue computation.
• The completion of a send or receive operation can be managed using the Test, Wait, and Cancel methods.

NON-BLOCKING COMMUNICATION

OVERLAP COMMUNICATION
• Example:Process1overlapsacomputationwithsendingthemessageand receiving the reply.
• The computation divides randNum by 10 and prints the result.

OVERLAP COMMUNICATION

程序代写 CS代考加微信: powcoder QQ: 1823890830 Email: powcoder@163.com

Related Posts