MPI communicators

class gpaw.mpi._Communicator(comm, parent=None)[source]

Construct a wrapper of the C-object for any MPI-communicator.

Parameters:

comm: MPI-communicator

Communicator.

Attributes:

size

Number of ranks in the MPI group.

rank

Number of this CPU in the MPI group.

parent

Parent MPI-communicator.

abort(errcode)[source]

Terminate MPI execution environment of all tasks in the group. This function only returns in the advent of an error occurring.

Parameters:

errcode: int

Error code to return to the invoking environment.

all_gather(a, b)[source]

Gather data from all ranks onto all processes in a group.

Parameters:

a: ndarray

Source of the data to gather, i.e. send buffer of this rank.

b: ndarray

Destination of the distributed data, i.e. receive buffer. The size of this array must match the size of the distributed source arrays multiplied by the number of process in the group.

Example:

# All ranks have parts of interesting data. Gather on all ranks.
seed = 123456
rng = np.random.default_rng(seed)
mydata = rng.normal(size=N)
data = np.empty(N*comm.size, dtype=float)
comm.all_gather(mydata, data)

# .. which is equivalent to ..

if comm.rank == 0:
    # Insert my part directly
    data[0:N] = mydata
    # Gather parts from the slaves
    buf = np.empty(N, dtype=float)
    for rank in range(1, comm.size):
        comm.receive(buf, rank, tag=123)
        data[rank*N:(rank+1)*N] = buf
else:
    # Send to the master
    comm.send(mydata, 0, tag=123)
# Broadcast from master to all slaves
comm.broadcast(data, 0)
alltoallv(sbuffer, scounts, sdispls, rbuffer, rcounts, rdispls)[source]

All-to-all in a group.

Parameters:

sbuffer: ndarray

Source of the data to distribute, i.e., send buffers on all ranks

scounts: ndarray

Integer array equal to the group size specifying the number of elements to send to each processor

sdispls: ndarray

Integer array (of length group size). Entry j specifies the displacement (relative to sendbuf from which to take the outgoing data destined for process j)

rbuffer: ndarray

Destination of the distributed data, i.e., local receive buffer.

rcounts: ndarray

Integer array equal to the group size specifying the maximum number of elements that can be received from each processor.

rdispls:

Integer array (of length group size). Entry i specifies the displacement (relative to recvbuf at which to place the incoming data from process i

barrier()[source]

Block execution until all process have reached this point.

broadcast(a, root)[source]

Share data from a single process to all ranks in a group.

Parameters:

a: ndarray

Data, i.e. send buffer on root rank, receive buffer elsewhere. Note that after the broadcast, all ranks have the same data.

root: int

Rank of the root process, from which the data is to be shared.

Example:

# All ranks have parts of interesting data. Take a given index.
seed = 123456
rng = np.random.default_rng(seed)
mydata[:] = rng.normal(size=N)

# Who has the element at global index 13? Everybody needs it!
index = 13
root, myindex = divmod(index, N)
element = np.empty(1, dtype=float)
if comm.rank == root:
    # This process has the requested element so extract it
    element[:] = mydata[myindex]

# Broadcast from owner to everyone else
comm.broadcast(element, root)

# .. which is equivalent to ..

if comm.rank == root:
    # We are root so send it to the other ranks
    for rank in range(comm.size):
        if rank != root:
            comm.send(element, rank, tag=123)
else:
    # We don't have it so receive from root
    comm.receive(element, root, tag=123)
compare(othercomm)[source]

Compare communicator to other.

Returns ‘ident’ if they are identical, ‘congruent’ if they are copies of each other, ‘similar’ if they are permutations of each other, and otherwise ‘unequal’.

This method corresponds to MPI_Comm_compare.

gather(a, root, b=None)[source]

Gather data from all ranks onto a single process in a group.

Parameters:

a: ndarray

Source of the data to gather, i.e. send buffer of this rank.

root: int

Rank of the root process, on which the data is to be gathered.

b: ndarray (ignored on all ranks different from root; default None)

Destination of the distributed data, i.e. root’s receive buffer. The size of this array must match the size of the distributed source arrays multiplied by the number of process in the group.

The reverse operation is scatter.

Example:

# All ranks have parts of interesting data. Gather it on master.
seed = 123456
rng = np.random.default_rng(seed)
mydata = rng.normal(size=N)
if comm.rank == 0:
    data = np.empty(N*comm.size, dtype=float)
else:
    data = None
comm.gather(mydata, 0, data)

# .. which is equivalent to ..

if comm.rank == 0:
    # Extract my part directly
    data[0:N] = mydata
    # Gather parts from the slaves
    buf = np.empty(N, dtype=float)
    for rank in range(1, comm.size):
        comm.receive(buf, rank, tag=123)
        data[rank*N:(rank+1)*N] = buf
else:
    # Send to the master
    comm.send(mydata, 0, tag=123)
get_c_object()[source]

Return the C-object wrapped by this debug interface.

Whenever a communicator object is passed to C code, that object must be a proper C-object - not e.g. this debug wrapper. For this reason. The C-communicator object has a get_c_object() implementation which returns itself; thus, always call comm.get_c_object() and pass the resulting object to the C code.

get_members()[source]

Return the subset of processes which are members of this MPI group in terms of the ranks they are assigned on the parent communicator. For the world communicator, this is all integers up to size.

Example:

>>> world.rank, world.size  
(3, 4)
>>> world.get_members()  
array([0, 1, 2, 3])
>>> comm = world.new_communicator(np.array([2, 3]))  
>>> comm.rank, comm.size  
(1, 2)
>>> comm.get_members()  
array([2, 3])
>>> comm.get_members()[comm.rank] == world.rank  
True
max(a, root=-1)[source]

Find maximal value by an MPI reduce operation of numerical data.

Parameters:

a: ndarray or value (type int or float)

Numerical data to find the maximum value of across all ranks in the communicator group. NB: Find global maximum from local max. If the data is a single value of type int or float (no complex), the result is returned because the input argument is immutable. Otherwise, the reduce operation is carried out in-place such that the elements of the input array will represent the max of the equivalent elements across all processes in the group.

root: int (default -1)

Rank of the root process, on which the outcome of the reduce operation is valid. A root rank of -1 signifies that the result will be distributed back to all processes, i.e. a broadcast.

min(a, root=-1)[source]

Find minimal value by an MPI reduce operation of numerical data.

Parameters:

a: ndarray or value (type int or float)

Numerical data to find the minimal value of across all ranks in the communicator group. NB: Find global minimum from local min. If the data is a single value of type int or float (no complex), the result is returned because the input argument is immutable. Otherwise, the reduce operation is carried out in-place such that the elements of the input array will represent the min of the equivalent elements across all processes in the group.

root: int (default -1)

Rank of the root process, on which the outcome of the reduce operation is valid. A root rank of -1 signifies that the result will be distributed back to all processes, i.e. a broadcast.

name()[source]

Return the name of the processor as a string.

new_communicator(ranks)[source]

Create a new MPI communicator for a subset of ranks in a group. Must be called with identical arguments by all relevant processes.

Note that a valid communicator is only returned to the processes which are included in the new group; other ranks get None returned.

Parameters:

ranks: ndarray (type int)

List of integers of the ranks to include in the new group. Note that these ranks correspond to indices in the current group whereas the rank attribute in the new communicators correspond to their respective index in the subset.

product(a, root=-1)[source]

Do multiplication by MPI reduce operations of numerical data.

Parameters:

a: ndarray or value (type int or float)

Numerical data to multiply across all ranks in the communicator group. NB: Find the global product from the local products. If the data is a single value of type int or float (no complex), the result is returned because the input argument is immutable. Otherwise, the reduce operation is carried out in-place such that the elements of the input array will represent the product of the equivalent elements across all processes in the group.

root: int (default -1)

Rank of the root process, on which the outcome of the reduce operation is valid. A root rank of -1 signifies that the result will be distributed back to all processes, i.e. a broadcast.

scatter(a, b, root)[source]

Distribute data from one rank to all other processes in a group.

Parameters:

a: ndarray (ignored on all ranks different from root; use None)

Source of the data to distribute, i.e. send buffer on root rank.

b: ndarray

Destination of the distributed data, i.e. local receive buffer. The size of this array multiplied by the number of process in the group must match the size of the source array on the root.

root: int

Rank of the root process, from which the source data originates.

The reverse operation is gather.

Example:

# The master has all the interesting data. Distribute it.
seed = 123456
rng = np.random.default_rng(seed)
if comm.rank == 0:
    data = rng.normal(size=N*comm.size)
else:
    data = None
mydata = np.empty(N, dtype=float)
comm.scatter(data, mydata, 0)

# .. which is equivalent to ..

if comm.rank == 0:
    # Extract my part directly
    mydata[:] = data[0:N]
    # Distribute parts to the slaves
    for rank in range(1, comm.size):
        buf = data[rank*N:(rank+1)*N]
        comm.send(buf, rank, tag=123)
else:
    # Receive from the master
    comm.receive(mydata, 0, tag=123)
sum(a, root=-1)[source]

Perform summation by MPI reduce operations of numerical data.

Parameters:

a: ndarray or value (type int, float or complex)

Numerical data to sum over all ranks in the communicator group. If the data is a single value of type int, float or complex, the result is returned because the input argument is immutable. Otherwise, the reduce operation is carried out in-place such that the elements of the input array will represent the sum of the equivalent elements across all processes in the group.

root: int (default -1)

Rank of the root process, on which the outcome of the reduce operation is valid. A root rank of -1 signifies that the result will be distributed back to all processes, i.e. a broadcast.

test(request)[source]

Test whether a non-blocking MPI operation has completed. A boolean is returned immediately and the request is not modified in any way.

Parameters:

request: MPI request

Request e.g. returned from send/receive when block=False is used.

testall(requests)[source]

Test whether non-blocking MPI operations have completed. A boolean is returned immediately but requests may have been deallocated as a result, provided they have completed before or during this invokation.

Parameters:

request: MPI request

Request e.g. returned from send/receive when block=False is used.

translate_ranks(other, ranks)[source]

“Translate ranks from communicator to other.

ranks must be valid on this communicator. Returns ranks on other communicator corresponding to the same processes. Ranks that are not defined on the other communicator are assigned values of -1. (In contrast to MPI which would assign MPI_UNDEFINED).

wait(request)[source]

Wait for a non-blocking MPI operation to complete before returning.

Parameters:

request: MPI request

Request e.g. returned from send/receive when block=False is used.

waitall(requests)[source]

Wait for non-blocking MPI operations to complete before returning.

Parameters:

requests: list

List of MPI requests e.g. aggregated from returned requests of multiple send/receive calls where block=False was used.

gpaw.mpi.send(obj, rank, comm)[source]

Send object to rank on the MPI communicator comm.

gpaw.mpi.receive(rank, comm)[source]

Receive object from rank on the MPI communicator comm.

gpaw.mpi.broadcast_array(array, *communicators)[source]

Broadcast np.ndarray across sequence of MPI-communicators.

gpaw.mpi.broadcast_exception(comm)[source]

Make sure all ranks get a possible exception raised.

This example:

with broadcast_exception(world):
    if world.rank == 0:
        x = 1 / 0

will raise ZeroDivisionError in the whole world.