D R A F T

Document for a Standard Message-Passing Interface

Message Passing Interface Forum

May 30, 1995

This work was supported in part by ARPA and NSF under grant ASC-9310330, the

National Science Foundation Science and Technology Center Cooperative Agreement No. CCR-8809615, and by the Commission of the European Community

through Esprit project P6643.

This is the result of a LaTeX run of a draft of a single chapter of the MPIF Final Report document.

Chapter 1

Groups, Contexts, and Communicators

1.1 Introduction

This chapter introduces MPI features that support the development of parallel libraries. Parallel libraries are needed to encapsulate the distracting complications inherent in parallel implementations of key algorithms. They help to ensure consistent correctness of such procedures, and provide a “higher level” of portability than MPI itself can provide. As such, libraries prevent each programmer from repeating the work of defining consistent data structures, data layouts, and methods that implement key algorithms (such as matrix operations). Since the best libraries come with several variations on parallel systems (different data layouts, different strategies depending on the size of the system or problem, or type of floating point), this too needs to be hidden from the user.

We refer the reader to [4] and [1] for further information on writing libraries in MPI, using the features described in this chapter.

1.1.1 Features Needed to Support Libraries

The key features needed to support the creation of robust parallel libraries are as follows:

• Safe communication space, that guarantees that libraries can communicate as they need to, without conflicting with communication extraneous to the library,

• Group scope for collective operations, that allow libraries to avoid unnecessarily synchronizing uninvolved processes (potentially running unrelated code),

• Abstract process naming to allow libraries to describe their communication in terms suitable to their own data structures and algorithms,

• The ability to “adorn” a set of communicating processes with additional user-defined attributes, such as extra collective operations. This mechanism should provide a means for the user or library writer effectively to extend a message-passing notation.

In addition, a unified mechanism or object is needed for conveniently denoting communication context, the group of communicating processes, to house abstract process naming, and to store adornments.

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

1.1.2 MPI’s Support for Libraries

The corresponding concepts that MPI provides, specifically to support robust libraries, are as follows:

• Contexts of communication,

• Groups of processes,

• Virtual topologies,

• Attribute caching,

• Communicators.

Communicators (see [2, 3, 5]) encapsulate all of these ideas in order to provide the appropriate scope for all communication operations in MPI. Communicators are divided into two kinds: intra-communicators for operations within a single group of processes, and intercommunicators, for point-to-point communication between two groups of processes.

Caching. Communicators (see below) provide a “caching” mechanism that allows one to associate new attributes with communicators, on a par with MPI built-in features. This can be used by advanced users to adorn communicators further, and by MPI to implement some communicator functions. For example, the virtual-topology functions described in Chapter ?? are likely to be supported this way.

Groups. Groups define an ordered collection of processes, each with a rank, and it is this group that defines the low-level names for inter-process communication (ranks are used for sending and receiving). Thus, groups define a scope for process names in point-to-point communication. In addition, groups define the scope of collective operations. Groups may be manipulated separately from communicators in MPI, but only communicators can be used in communication operations.

Intra-communicators. The most commonly used means for message passing in MPI is via intra-communicators. Intra-communicators contain an instance of a group, contexts of communication for both point-to-point and collective communication, and the ability to include virtual topology and other attributes. These features work as follows:

• Contexts provide the ability to have separate safe “universes” of message passing in MPI. A context is akin to an additional tag that differentiates messages. The system manages this differentiation process. The use of separate communication contexts by distinct libraries (or distinct library invocations) insulates communication internal to the library execution from external communication. This allows the invocation of the library even if there are pending communications on “other” communicators, and avoids the need to synchronize entry or exit into library code. Pending point-to-point communications are also guaranteed not to interfere with collective communications within a single communicator.

• Groups define the participants in the communication (see above) of a communicator.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

• A virtual topology defines a special mapping of the ranks in a group to and from a topology. Special constructors for communicators are defined in chapter ?? to provide this feature. Intra-communicators as described in this chapter do not have topologies.

• Attributes define the local information that the user or library has added to a communicator for later reference.

Advice to users. The current practice in many communication libraries is that there is a unique, predefined communication universe that includes all processes available when the parallel program is initiated; the processes are assigned consecutive ranks. Participants in a point-to-point communication are identified by their rank; a collective communication (such as broadcast) always involves all processes. This practice can be followed in MPI by using the predefined communicator MPI COMM WORLD. Users who are satisfied with this practice can plug in MPI COMM WORLD wherever a communicator argument is required, and can consequently disregard the rest of this chapter. (End of advice to users.)

Inter-communicators. The discussion has dealt so far with intra-communication: communication within a group. MPI also supports inter-communication: communication between two non-overlapping groups. When an application is built by composing several parallel modules, it is convenient to allow one module to communicate with another using local ranks for addressing within the second module. This is especially convenient in a client-server computing paradigm, where either client or server are parallel. The support of inter-communication also provides a mechanism for the extension of MPI to a dynamic model where not all processes are preallocated at initialization time. In such a situation, it becomes necessary to support communication across “universes.” Inter-communication is supported by objects called inter-communicators. These objects bind two groups together with communication contexts shared by both groups. For inter-communicators, these features work as follows:

• Contexts provide the ability to have a separate safe “universe” of message passing between the two groups. A send in the local group is always a receive in the remote group, and vice versa. The system manages this differentiation process. The use of separate communication contexts by distinct libraries (or distinct library invocations) insulates communication internal to the library execution from external communication. This allows the invocation of the library even if there are pending communications on “other” communicators, and avoids the need to synchronize entry or exit into library code. There is no general-purpose collective communication on inter-communicators, so contexts are used just to isolate point-to-point communication.

• A local and remote group specify the recipients and destinations for an inter-com-municator.

• Virtual topology is undefined for an inter-communicator.

• As before, attributes cache defines the local information that the user or library has added to a communicator for later reference.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI provides mechanisms for creating and manipulating inter-communicators. They are used for point-to-point communication in an related manner to intra-communicators. Users who do not need inter-communication in their applications can safely ignore this extension. Users who need collective operations via inter-communicators must layer it on top of MPI. Users who require inter-communication between overlapping groups must also layer this capability on top of MPI.

1.2 Basic Concepts

In this section, we turn to a more formal definition of the concepts introduced above.

1.2.1 Groups

A group is an ordered set of process identifiers (henceforth processes); processes are implementation-dependent objects. Each process in a group is associated with an integer rank. Ranks are contiguous and start from zero. Groups are represented by opaque group objects, and hence cannot be directly transferred from one process to another. A group is used within a communicator to describe the participants in a communication “universe” and to rank such participants (thus giving them unique names within that “universe” of communication).

There is a special pre-defined group: MPI GROUP EMPTY, which is a group with no members. The predefined constant MPI GROUP NULL is the value used for invalid group handles.

Advice to users. MPI GROUP EMPTY, which is a valid handle to an empty group, should not be confused with MPI GROUP NULL, which in turn is an invalid handle. The former may be used as an argument to group operations; the latter, which is returned when a group is freed, in not a valid argument. (End of advice to users.)

Advice to implementors. A group may be represented by a virtual-to-real processaddress-translation table. Each communicator object (see below) would have a pointer to such a table.

Simple implementations of MPI will enumerate groups, such as in a table. However, more advanced data structures make sense in order to improve scalability and memory usage with large numbers of processes. Such implementations are possible with MPI. (End of advice to implementors.)

1.2.2 Contexts

A context is a property of communicators (defined next) that allows partitioning of the communication space. A message sent in one context cannot be received in another context. Furthermore, where permitted, collective operations are independent of pending point-to-point operations. Contexts are not explicit MPI objects; they appear only as part of the realization of communicators (below).

Advice to implementors. Distinct communicators in the same process have distinct contexts. A context is essentially a system-managed tag (or tags) needed to make a communicator safe for point-to-point and MPI-defined collective communication.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Safety means that collective and point-to-point communication within one communicator do not interfere, and that communication over distinct communicators don’t interfere. A possible implementation for a context is as a supplemental tag attached to messages on send and matched on receive. Each intra-communicator stores the value of its two tags (one for point-to-point and one for collective communication). Communicatorgenerating functions use a collective communication to agree on a new group-wide unique context. Analogously, in inter-communication (which is strictly point-to-point communication), two context tags are stored per communicator, one used by group A to send and group B to receive, and a second used by group B to send and for group A to receive. Since contexts are not explicit objects, other implementations are also possible. (End of advice to implementors.)

1.2.3 Intra-Communicators

Intra-communicators bring together the concepts of group and context. To support implementation-specific optimizations, and application topologies (defined in the next chapter, chapter ??), communicators may also “cache” additional information (see section 1.7). MPI communication operations reference communicators to determine the scope and the “communication universe” in which a point-to-point or collective operation is to operate.

Each communicator contains a group of valid participants; this group always includes the local process. The source and destination of a message is identified by process rank within that group.

For collective communication, the intra-communicator specifies the set of processes that participate in the collective operation (and their order, when significant). Thus, the communicator restricts the “spatial” scope of communication, and provides machine-independent process addressing through ranks.

Intra-communicators are represented by opaque intra-communicator objects, and hence cannot be directly transferred from one process to another.

1.2.4 Predefined Intra-Communicators

An initial intra-communicator MPI COMM WORLD of all processes the local process can communicate with after initialization (itself included) is defined once MPI INIT has been called. In addition, the communicator MPI COMM SELF is provided, which includes only the process itself.

The predefined constant MPI COMM NULL is the value used for invalid communicator handles.

In a static-process-model implementation of MPI, all processes that participate in the computation are available after MPI is initialized. For this case, MPI COMM WORLD is a communicator of all processes available for the computation; this communicator has the same value in all processes. In an implementation of MPI where processes can dynamically join an MPI execution, it may be the case that a process starts an MPI computation without having access to all other processes. In such situations, MPI COMM WORLD is a communicator incorporating all processes with which the joining process can immediately communicate. Therefore, MPI COMM WORLD may simultaneously have different values in different processes.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

All MPI implementations are required to provide the MPI COMM WORLD communicator. It cannot be deallocated during the life of a process. The group corresponding to this communicator does not appear as a pre-defined constant, but it may be accessed using MPI COMM GROUP (see below). MPI does not specify the correspondence between the process rank in MPI COMM WORLD and its (machine-dependent) absolute address. Neither does MPI specify the function of the host process, if any. Other implementation-dependent, predefined communicators may also be provided.

1.3 Group Management

This section describes the manipulation of process groups in MPI. These operations are local and their execution do not require interprocess communication.

1.3.1 Group Accessors

MPI GROUP SIZE(group, size)

IN  group  group (handle)
OUT  size  number of processes in the group (integer)

int MPI Group size(MPI Group group, int *size)
MPI GROUP SIZE(GROUP, SIZE, IERROR)
    INTEGER GROUP, SIZE, IERROR

MPI GROUP RANK(group, rank)

IN  group  group (handle)
OUT  rank  rank of the calling process in group, or

MPI UNDEFINED if the process is not a member (integer)

int MPI Group rank(MPI Group group, int *rank)
MPI GROUP RANK(GROUP, RANK, IERROR)
    INTEGER GROUP, RANK, IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI GROUP TRANSLATE RANKS (group1, n, ranks1, group2, ranks2)

IN  group1  group1 (handle)
IN  n  number of ranks in ranks1 and ranks2 arrays (integer)
IN  ranks1  array of zero or more valid ranks in group1
IN  group2  group2 (handle)
OUT  ranks2  array of corresponding ranks in group2, MPI UNDE-

FINED when no correspondence exists.

int MPI Group translate ranks (MPI Group group1, int n, int *ranks1,
             MPI Group group2, int *ranks2)
MPI GROUP TRANSLATE RANKS(GROUP1, N, RANKS1, GROUP2, RANKS2, IERROR)
    INTEGER GROUP1, N, RANKS1(*), GROUP2, RANKS2(*), IERROR

This function is important for determining the relative numbering of the same processes in two different groups. For instance, if one knows the ranks of certain processes in the group of MPI COMM WORLD, one might want to know their ranks in a subset of that group.

MPI GROUP COMPARE(group1, group2, result)

IN  group1  first group (handle)
IN  group2  second group (handle)
OUT  result  result (integer)

int MPI Group compare(MPI Group group1,MPI Group group2, int *result)
MPI GROUP COMPARE(GROUP1, GROUP2, RESULT, IERROR)
    INTEGER GROUP1, GROUP2, RESULT, IERROR

MPI IDENT results if the group members and group order is exactly the same in both groups. This happens for instance if group1 and group2 are the same handle. MPI SIMILAR results if the group members are the same but the order is different. MPI UNEQUAL results otherwise.

1.3.2 Group Constructors

Group constructors are used to subset and superset existing groups. These constructors construct new groups from existing groups. These are local operations, and distinct groups may be defined on different processes; a process may also define a group that does not include itself. Consistent definitions are required when groups are used as arguments in communicator-building functions. MPI does not provide a mechanism to build a group from scratch, but only from other, previously defined groups. The base group, upon which all other groups are defined, is the group associated with the initial communicator MPI COMM WORLD (accessible through the function MPI COMM GROUP).

Rationale. In what follows, there is no group duplication function analogous to MPI COMM DUP, defined later in this chapter. There is no need for a group duplicator. A group, once created, can have several references to it by making copies of

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

the handle. The following constructors address the need for subsets and supersets of existing groups. (End of rationale.)

Advice to implementors. Each group constructor behaves as if it returned a new group object. When this new group is a copy of an existing group, then one can avoid creating such new objects, using a reference-count mechanism. (End of advice to implementors.)

MPI COMM GROUP(comm, group)

IN  comm  communicator (handle)
OUT  group  group corresponding to comm (handle)

int MPI Comm group(MPI Comm comm, MPI Group *group)
MPI COMM GROUP(COMM, GROUP, IERROR)
    INTEGER COMM, GROUP, IERROR

MPI COMM GROUP returns in group a handle to the group of comm.

MPI GROUP UNION(group1, group2, newgroup)

IN  group1  first group (handle)
IN  group2  second group (handle)
OUT  newgroup  union group (handle)

int MPI Group union(MPI Group group1, MPI Group group2, MPI Group *newgroup)
MPI GROUP UNION(GROUP1, GROUP2, NEWGROUP, IERROR)
    INTEGER GROUP1, GROUP2, NEWGROUP, IERROR

MPI GROUP INTERSECTION(group1, group2, newgroup)

IN  group1  first group (handle)
IN  group2  second group (handle)
OUT  newgroup  intersection group (handle)

int MPI Group intersection(MPI Group group1, MPI Group group2,
             MPI Group *newgroup)
MPI GROUP INTERSECTION(GROUP1, GROUP2, NEWGROUP, IERROR)
    INTEGER GROUP1, GROUP2, NEWGROUP, IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI GROUP DIFFERENCE(group1, group2, newgroup)

IN  group1  first group (handle)
IN  group2  second group (handle)
OUT  newgroup  difference group (handle)

int MPI Group difference(MPI Group group1, MPI Group group2,
             MPI Group *newgroup)
MPI GROUP DIFFERENCE(GROUP1, GROUP2, NEWGROUP, IERROR)
    INTEGER GROUP1, GROUP2, NEWGROUP, IERROR

The set-like operations are defined as follows:

union All elements of the first group (group1), followed by all elements of second group (group2) not in first.

intersect all elements of the first group that are also in the second group, ordered as in first group.

difference all elements of the first group that are not in the second group, ordered as in the first group.

Note that for these operations the order of processes in the output group is determined primarily by order in the first group (if possible) and then, if necessary, by order in the second group. Neither union nor intersection are commutative, but both are associative.

The new group can be empty, that is, equal to MPI GROUP EMPTY.

MPI GROUP INCL(group, n, ranks, newgroup)

IN  group  group (handle)
IN  n  number of elements in array ranks (and size of new-

group) (integer)

IN  ranks  ranks of processes in group to appear in newgroup (ar-

ray of integers)

OUT  newgroup  new group derived from above, in the order defined by

ranks (handle)

int MPI Group incl(MPI Group group, int n, int *ranks, MPI Group *newgroup)
MPI GROUP INCL(GROUP, N, RANKS, NEWGROUP, IERROR)
    INTEGER GROUP, N, RANKS(*), NEWGROUP, IERROR

The function MPI GROUP INCL creates a group newgroup that consists of the n processes in group with ranks rank[0],…, rank[n-1]; the process with rank i in newgroup is the process with rank ranks[i] in group. Each of the n elements of ranks must be a valid rank in group and all elements must be distinct, or else the program is erroneous. If n = 0, then newgroup is MPI GROUP EMPTY. This function can, for instance, be used to reorder the elements of a group. See also MPI GROUP COMPARE.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI GROUP EXCL(group, n, ranks, newgroup)

IN  group  group (handle)
IN  n  number of elements in array ranks (integer)
IN  ranks  array of integer ranks in group not to appear in new-

group

OUT  newgroup  new group derived from above, preserving the order

defined by group (handle)

int MPI Group excl(MPI Group group, int n, int *ranks, MPI Group *newgroup)
MPI GROUP EXCL(GROUP, N, RANKS, NEWGROUP, IERROR)
    INTEGER GROUP, N, RANKS(*), NEWGROUP, IERROR

The function MPI GROUP EXCL creates a group of processes newgroup that is obtained by deleting from group those processes with ranks ranks[0] ,… ranks[n-1]. The ordering of processes in newgroup is identical to the ordering in group. Each of the n elements of ranks must be a valid rank in group and all elements must be distinct; otherwise, the program is erroneous. If n = 0, then newgroup is identical to group.

MPI GROUP RANGE INCL(group, n, ranges, newgroup)

IN  group  group (handle)
IN  n  number of triplets in array ranges (integer)
IN  ranges  an array of integer triplets, of the form (first rank, last

rank, stride) indicating ranks in group of processes to be included in newgroup

OUT  newgroup  new group derived from above, in the order defined by

ranges (handle)

int MPI Group range incl(MPI Group group, int n, int ranges[][3],
             MPI Group *newgroup)
MPI GROUP RANGE INCL(GROUP, N, RANGES, NEWGROUP, IERROR)
    INTEGER GROUP, N, RANGES(3,*), NEWGROUP, IERROR

If ranges consist of the triplets

(first_{1}, last_{1}, stride_{1}), ..., (first_{n}, last_{n}, stride_{n})

then newgroup consists of the sequence of processes in group with ranks

first_{1}, first_{1} + stride_{1}, ..., first_{1} + ⌊\frac{last_{1} − first_{1}}{stride_{1}} ⌋ stride_{1}, ...

first_{n}, first_{n} + stride_{n}, ..., first_{n} + ⌊\frac{last_{n} − first_{n}}{stride_{n}} ⌋ stride_{n}.

Each computed rank must be a valid rank in group and all computed ranks must be distinct, or else the program is erroneous. Note that we may have first_{i} > last_{i}, and stride_{i} may be negative, but cannot be zero.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

The functionality of this routine is specified to be equivalent to expanding the array of ranges to an array of the included ranks and passing the resulting array of ranks and other arguments to MPI GROUP INCL. A call to MPI GROUP INCL is equivalent to a call to MPI GROUP RANGE INCL with each rank i in ranks replaced by the triplet (i,i,1) in the argument ranges.

MPI GROUP RANGE EXCL(group, n, ranges, newgroup)

IN  group  group (handle)
IN  n  number of elements in array ranks (integer)
IN  ranges  a one-dimensional array of integer triplets of the form

(first rank, last rank, stride), indicating the ranks in group of processes to be excluded from the output group newgroup.

OUT  newgroup  new group derived from above, preserving the order

in group (handle)

int MPI Group range excl(MPI Group group, int n, int ranges[][3],
             MPI Group *newgroup)
MPI GROUP RANGE EXCL(GROUP, N, RANGES, NEWGROUP, IERROR)
    INTEGER GROUP, N, RANGES(3,*), NEWGROUP, IERROR

Each computed rank must be a valid rank in group and all computed ranks must be distinct, or else the program is erroneous.

The functionality of this routine is specified to be equivalent to expanding the array of ranges to an array of the excluded ranks and passing the resulting array of ranks and other arguments to MPI GROUP EXCL. A call to MPI GROUP EXCL is equivalent to a call to MPI GROUP RANGE EXCL with each rank i in ranks replaced by the triplet (i,i,1) in the argument ranges.

Advice to users. The range operations do not explicitly enumerate ranks, and therefore are more scalable if implemented efficiently. Hence, we recommend MPI programmers to use them whenenever possible, as high-quality implementations will take advantage of this fact. (End of advice to users.)

Advice to implementors. The range operations should be implemented, if possible, without enumerating the group members, in order to obtain better scalability (time and space). (End of advice to implementors.)

1.3.3 Group Destructors

MPI GROUP FREE(group)

INOUT group group (handle)

int MPI Group free(MPI Group *group)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI GROUP FREE(GROUP, IERROR)
INTEGER GROUP, IERROR

This operation marks a group object for deallocation. The handle group is set to MPI GROUP NULL by the call. Any on-going operation using this group will complete normally.

Advice to implementors. One can keep a reference count that is incremented for each call to MPI COMM CREATE and MPI COMM DUP, and decremented for each call to MPI GROUP FREE or MPI COMM FREE; the group object is ultimately deallocated when the reference count drops to zero. (End of advice to implementors.)

1.4 Communicator Management

This section describes the manipulation of communicators in MPI. Operations that access communicators are local and their execution does not require interprocess communication. Operations that create communicators are collective and may require interprocess communication.

Advice to implementors. High-quality implementations should amortize the overheads associated with the creation of communicators (for the same group, or subsets thereof) over several calls, by allocating multiple contexts with one collective communication. (End of advice to implementors.)

1.4.1 Communicator Accessors

The following are all local operations.

MPI COMM SIZE(comm, size)

IN  comm  communicator (handle)
OUT  size  number of processes in the group of comm (integer)

int MPI Comm size(MPI Comm comm, int *size)
MPI COMM SIZE(COMM, SIZE, IERROR)
INTEGER COMM, SIZE, IERROR

Rationale. This function is equivalent to accessing the communicator’s group with MPI COMM GROUP (see below), computing the size using MPI GROUP SIZE, and then freeing the group temporary via MPI GROUP FREE. However, this function is so commonly used, that this shortcut was introduced. (End of rationale.)

Advice to users. This function indicates the number of processes involved in a communicator. For MPI COMM WORLD, it indicates the total number of processes available (for this version of MPI, there is no standard way to change the number of processes once initialization has taken place).

This call is often used with the next call to determine the amount of concurrency available for a specific library or program. The following call, MPI COMM RANK

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

indicates the rank of the process that calls it in the range from 0 …size−1, where size is the return value of MPI COMM SIZE.(End of advice to users.)

MPI COMM RANK(comm, rank)

IN  comm  communicator (handle)
OUT  rank  rank of the calling process in group of comm (integer)

int MPI Comm rank(MPI Comm comm, int *rank)
MPI COMM RANK(COMM, RANK, IERROR)
    INTEGER COMM, RANK, IERROR

⊤

Rationale. This function is equivalent to accessing the communicator’s group with MPI COMM GROUP (see above), and then computing the rank using MPI GROUP RANK. (End of rationale.)

⊥

Advice to users. This function gives the rank of the process in the particular communicator’s group. It is useful, as noted above, in conjunction with MPI COMM SIZE.

Many programs will be written with the master-slave model, where one process (such as the rank-zero process) will play a supervisory role, and the other processes will serve as compute nodes. In this framework, the two preceding calls are useful for determining the roles of the various processes of a communicator. (End of advice to users.)

MPI COMM COMPARE(comm1, comm2, result)

IN  comm1  first communicator (handle)
IN  comm2  second communicator (handle)
OUT  result  result (integer)

int MPI Comm compare(MPI Comm comm1,MPI Comm comm2, int *result)
MPI COMM COMPARE(COMM1, COMM2, RESULT, IERROR)
    INTEGER COMM1, COMM2, RESULT, IERROR

MPI IDENT results if and only if comm1 and comm2 are handles for the same object (identical groups and same contexts). MPI CONGRUENT results if the underlying groups are identical in constituents and rank order; these communicators differ only by context. MPI SIMILAR results if the group members of both communicators are the same but the rank order differs. MPI UNEQUAL results otherwise.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

1.4.2 Communicator Constructors

The following are collective functions that are invoked by all processes in the group associated with comm.

Rationale. Note that there is a chicken-and-egg aspect to MPI in that a communicator is needed to create a new communicator. The base communicator for all MPI communicators is predefined outside of MPI, and is MPI COMM WORLD. This model was arrived at after considerable debate, and was chosen to increase “safety” of programs written in MPI. (End of rationale.)

MPI COMM DUP(comm, newcomm)

IN  comm  communicator (handle)
OUT  newcomm  copy of comm (handle)

int MPI Comm dup(MPI Comm comm, MPI Comm *newcomm)
MPI COMM DUP(COMM, NEWCOMM, IERROR)
INTEGER COMM, NEWCOMM, IERROR

MPI COMM DUP Duplicates the existing communicator comm with associated key values. For each key value, the respective copy callback function determines the attribute value associated with this key in the new communicator; one particular action that a copy callback may take is to delete the attribute from the new communicator. Returns in newcomm a new communicator with the same group, any copied cached information, but a new context (see section 1.7.1).

Advice to users. This operation is used to provide a parallel library call with a duplicate communication space that has the same properties as the original communicator. This includes any attributes (see below), and topologies (see chapter ??). This call is valid even if there are pending point-to-point communications involving the communicator comm. A typical call might involve a MPI COMM DUP at the beginning of the parallel call, and an MPI COMM FREE of that duplicated communicator at the end of the call. Other models of communicator management are also possible. This call applies to both intra- and inter-communicators. (End of advice to users.)

Advice to implementors. One need not actually copy the group information, but only add a new reference and increment the reference count. Copy on write can be used for the cached information.(End of advice to implementors.)

MPI COMM CREATE(comm, group, newcomm)

IN  comm  communicator (handle)
IN  group  Group, which is a subset of the group of comm (han-

dle)

OUT  newcomm  new communicator (handle)

int MPI Comm create(MPI Comm comm, MPI Group group, MPI Comm *newcomm)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI COMM CREATE(COMM, GROUP, NEWCOMM, IERROR)
INTEGER COMM, GROUP, NEWCOMM, IERROR

This function creates a new communicator newcomm with communication group defined by group and a new context. No cached information propagates from comm to newcomm. The function returns MPI COMM NULL to processes that are not in group. The call is erroneous if not all group arguments have the same value, or if group is not a subset of the group associated with comm. Note that the call is to be executed by all processes in comm, even if they do not belong to the new group. This call applies only to intra-communicators.

Rationale. The requirement that the entire group of comm participate in the call stems from the following considerations:

• It allows the implementation to layer MPI COMM CREATE on top of regular collective communications.

• It provides additional safety, in particular in the case where partially overlapping groups are used to create new communicators.

• It permits implementations sometimes to avoid communication related to context creation.

(End of rationale.)

Advice to users. MPI COMM CREATE provides a means to subset a group of processes for the purpose of separate MIMD computation, with separate communication space. newcomm, which emerges from MPI COMM CREATE can be used in subsequent calls to MPI COMM CREATE (or other communicator constructors) further to subdivide a computation into parallel sub-computations. A more general service is provided by MPI COMM SPLIT, below. (End of advice to users.)

Advice to implementors. Since all processes calling MPI COMM DUP or MPI COMM CREATE provide the same group argument, it is theoretically possible to agree on a group-wide unique context with no communication. However, local execution of these functions requires use of a larger context name space and reduces error checking. Implementations may strike various compromises between these conflicting goals, such as bulk allocation of multiple contexts in one collective operation.

Important: If new communicators are created without synchronizing the processes involved then the communication system should be able to cope with messages arriving in a context that has not yet been allocated at the receiving process. (End of advice to implementors.)

MPI COMM SPLIT(comm, color, key, newcomm)

IN  comm  communicator (handle)
IN  color  control of subset assignment (integer)
IN  key  control of rank assigment (integer)
OUT  newcomm  new communicator (handle)

int MPI Comm split(MPI Comm comm, int color, int key, MPI Comm *newcomm)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI COMM SPLIT(COMM, COLOR, KEY, NEWCOMM, IERROR)
INTEGER COMM, COLOR, KEY, NEWCOMM, IERROR

This function partitions the group associated with comm into disjoint subgroups, one for each value of color. Each subgroup contains all processes of the same color. Within each subgroup, the processes are ranked in the order defined by the value of the argument key, with ties broken according to their rank in the old group. A new communicator is created for each subgroup and returned in newcomm. A process may supply the color value MPI UNDEFINED, in which case newcomm returns MPI COMM NULL. This is a collective call, but each process is permitted to provide different values for color and key.

A call to MPI COMM CREATE(comm, group, newcomm) is equivalent to a call to MPI COMM SPLIT(comm, color, key, newcomm), where all members of group provide color = 0 and key = rank in group, and all processes that are not members of group provide color = MPI UNDEFINED. The function MPI COMM SPLIT allows more general partitioning of a group into one or more subgroups with optional reordering. This call

⊤ applies only intra-communicators. ⊥ The value of color should be nonnegative

Advice to users. This is an extremely powerful mechanism for dividing a single communicating group of processes into k subgroups, with k chosen implicitly by the user (by the number of colors asserted over all the processes). Each resulting communicator will be non-overlapping. Such a division could be useful for defining a hierarchy of computations, such as for multigrid, or linear algebra.

Multiple calls to MPI COMM SPLIT can be used to overcome the requirement that any call have no overlap of the resulting communicators (each process is of only one color per call). In this way, multiple overlapping communication structures can be created. Creative use of the color and key in such splitting operations is encouraged.

Note that, for a fixed color, the keys need not be unique. It is MPI COMM SPLIT’s responsibility to sort processes in ascending order according to this key, and to break ties in a consistent way. If all the keys are specified in the same way, then all the processes in a given color will have the relative rank order as they did in their parent group. (In general, they will have different ranks.)

Essentially, making the key value zero for all processes of a given color means that one doesn’t really care about the rank-order of the processes in the new communicator. (End of advice to users.)

⊤

Rationale. color is restricted to be nonnegative, so as not to confict with the value assigned to MPI UNDEFINED. (End of rationale.)

⊥

1.4.3 Communicator Destructors

MPI COMM FREE(comm)

INOUT comm communicator to be destroyed (handle)

int MPI Comm free(MPI Comm *comm)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI COMM FREE(COMM, IERROR)
    INTEGER COMM, IERROR

This collective operation marks the communication object for deallocation. The handle is set to MPI COMM NULL. Any pending operations that use this communicator will complete normally; the object is actually deallocated only if there are no other active references to it. This call applies to intra- and inter-communicators. The delete callback functions for all cached attributes (see section 1.7) are called in arbitrary order.

Advice to implementors. A reference-count mechanism may be used: the reference count is incremented by each call to MPI COMM DUP, and decremented by each call to MPI COMM FREE. The object is ultimately deallocated when the count reaches zero.

Though collective, it is anticipated that this operation will normally be implemented to be local, though the debugging version of an MPI library might choose to synchronize. (End of advice to implementors.)

1.5 Motivating Examples

1.5.1 Current Practice #1

Example #1a:

   main(int argc, char **argv)
   {
     int me, size;
     ...
     MPI_Init ( &argc, &argv );
     MPI_Comm_rank (MPI_COMM_WORLD, &me);
     MPI_Comm_size (MPI_COMM_WORLD, &size);
     (void)printf ("Process %d size %d\n", me, size);
     ...
     MPI_Finalize();
   }

Example #1a is a do-nothing program that initializes itself legally, and refers to the the “all” communicator, and prints a message. It terminates itself legally too. This example does not imply that MPI supports printf-like communication itself. Example #1b (supposing that size is even):

    main(int argc, char **argv)
    {
       int me, size;
       int SOME_TAG = 0;
       ...
       MPI_Init(&argc, &argv);
       MPI_Comm_rank(MPI_COMM_WORLD, &me);   /* local */

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

  MPI_Comm_size(MPI_COMM_WORLD, &size); /* local */
  if((me % 2) == 0)
  {
     /* send unless highest-numbered process */
     if((me + 1) < size)
        MPI_Send(..., me + 1, SOME_TAG, MPI_COMM_WORLD);
  }
  else
     MPI_Recv(..., me - 1, SOME_TAG, MPI_COMM_WORLD);
  ...
  MPI_Finalize();
}

Example #1b schematically illustrates message exchanges between “even” and “odd” processes in the “all” communicator.

1.5.2 Current Practice #2

main(int argc, char **argv)
{
int me, count;
void *data;
...
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &me);
if(me == 0)
{
    /* get input, create buffer ``data'' */
    ...
}
MPI_Bcast(data, count, MPI_BYTE, 0, MPI_COMM_WORLD);
...
MPI_Finalize();
}

This example illustrates the use of a collective communication.

1.5.3 (Approximate) Current Practice #3

main(int argc, char **argv)
{
int me, count, count2;

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

void *send_buf, *recv_buf, *send_buf2, *recv_buf2;
MPI_Group MPI_GROUP_WORLD, grprem;
MPI_Comm commslave;
static int ranks[] = {0};
...
MPI_Init(&argc, &argv);
MPI_Comm_group(MPI_COMM_WORLD, &MPI_GROUP_WORLD);
MPI_Comm_rank(MPI_COMM_WORLD, &me);  /* local */
MPI_Group_excl(MPI_GROUP_WORLD, 1, ranks, &grprem);  /* local */
MPI_Comm_create(MPI_COMM_WORLD, grprem, &commslave);
if(me != 0)
{
  /* compute on slave */
  ...
  MPI_Reduce(send_buf,recv_buff,count, MPI_INT, MPI_SUM, 1, commslave);
  ...
}
/* zero falls through immediately to this reduce, others do later... */
MPI_Reduce(send_buf2, recv_buff2, count2,
           MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD);
MPI_Comm_free(&commslave);
MPI_Group_free(&MPI_GROUP_WORLD);
MPI_Group_free(&grprem);
MPI_Finalize();
}

This example illustrates how a group consisting of all but the zeroth process of the “all” group is created, and then how a communicator is formed ( commslave) for that new group. The new communicator is used in a collective call, and all processes execute a collective call in the MPI COMM WORLD context. This example illustrates how the two communicators (that inherently possess distinct contexts) protect communication. That is, communication in MPI COMM WORLD is insulated from communication in commslave, and vice versa.

In summary, “group safety” is achieved via communicators because distinct contexts within communicators are enforced to be unique on any process.

1.5.4 Example #4

The following example is meant to illustrate “safety” between point-to-point and collective communication. MPI guarantees that a single communicator can do safe point-to-point and collective communication.

#define TAG_ARBITRARY 12345
#define SOME_COUNT       50
main(int argc, char **argv)
{

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

int me;
MPI_Request request[2];
MPI_Status status[2];
MPI_Group MPI_GROUP_WORLD, subgroup;
int ranks[] = {2, 4, 6, 8};
MPI_Comm the_comm;
...
MPI_Init(&argc, &argv);
MPI_Comm_group(MPI_COMM_WORLD, &MPI_GROUP_WORLD);
MPI_Group_incl(MPI_GROUP_WORLD, 4, ranks, &subgroup); /* local */
MPI_Group_rank(subgroup, &me);     /* local */
MPI_Comm_create(MPI_COMM_WORLD, subgroup, &the_comm);
if(me != MPI_UNDEFINED)
{
    MPI_Irecv(buff1, count, MPI_DOUBLE, MPI_ANY_SOURCE, TAG_ARBITRARY,
                      the_comm, request);
    MPI_Isend(buff2, count, MPI_DOUBLE, (me+1)%4, TAG_ARBITRARY,
                      the_comm, request+1);
}
for(i = 0; i < SOME_COUNT, i++)
  MPI_Reduce(..., the_comm);
MPI_Waitall(2, request, status);
MPI_Comm_free(t&he_comm);
MPI_Group_free(&MPI_GROUP_WORLD);
MPI_Group_free(&subgroup);
MPI_Finalize();
}

1.5.5 Library Example #1

The main program:

main(int argc, char **argv)
{
int done = 0;
user_lib_t *libh_a, *libh_b;
void *dataset1, *dataset2;
...
MPI_Init(&argc, &argv);
...
init_user_lib(MPI_COMM_WORLD, &libh_a);
init_user_lib(MPI_COMM_WORLD, &libh_b);
...
user_start_op(libh_a, dataset1);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

user_start_op(libh_b, dataset2);
...
while(!done)
{
   /* work */
   ...
   MPI_Reduce(..., MPI_COMM_WORLD);
   ...
   /* see if done */
   ...
}
user_end_op(libh_a);
user_end_op(libh_b);
uninit_user_lib(libh_a);
uninit_user_lib(libh_b);
MPI_Finalize();
}

The user library initialization code:

void init_user_lib(MPI_Comm comm, user_lib_t **handle)
{
user_lib_t *save;
user_lib_initsave(&save); /* local */
MPI_Comm_dup(comm, &(save -> comm));
/* other inits */
...
*handle = save;
}

User start-up code:

void user_start_op(user_lib_t *handle, void *data)
{
MPI_Irecv( ..., handle->comm, &(handle -> irecv_handle) );
MPI_Isend( ..., handle->comm, &(handle -> isend_handle) );
}

User communication clean-up code:

void user_end_op(user_lib_t *handle)
{
MPI_Status *status;
MPI_Wait(handle -> isend_handle, status);
MPI_Wait(handle -> irecv_handle, status);
}

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

User object clean-up code:

void uninit_user_lib(user_lib_t *handle)
{
MPI_Comm_free(&(handle -> comm));
free(handle);
}

1.5.6 Library Example #2

The main program:

main(int argc, char **argv)
{
int ma, mb;
MPI_Group MPI_GROUP_WORLD, group_a, group_b;
MPI_Comm comm_a, comm_b;
static int list_a[] = {0, 1};
#if  defined(EXAMPLE_2B) | defined(EXAMPLE_2C)
static int list_b[] = {0, 2 ,3};
#else/* EXAMPLE_2A */
static int list_b[] = {0, 2};
#endif
int size_list_a = sizeof(list_a)/sizeof(int);
int size_list_b = sizeof(list_b)/sizeof(int);
...
MPI_Init(&argc, &argv);
MPI_Comm_group(MPI_COMM_WORLD, &MPI_GROUP_WORLD);
MPI_Group_incl(MPI_GROUP_WORLD, size_list_a, list_a, &group_a);
MPI_Group_incl(MPI_GROUP_WORLD, size_list_b, list_b, &group_b);
MPI_Comm_create(MPI_COMM_WORLD, group_a, &comm_a);
MPI_Comm_create(MPI_COMM_WORLD, group_b, &comm_b);
if(comm_a != MPI_COMM_NULL)
   MPI_Comm_rank(comm_a, &ma);
if(comm_a != MPI_COMM_NULL)
   MPI_Comm_rank(comm_b, &mb);
if(comm_a != MPI_COMM_NULL)
   lib_call(comm_a);
if(comm_b != MPI_COMM_NULL)
{
  lib_call(comm_b);
  lib_call(comm_b);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

     }
     if(comm_a != MPI_COMM_NULL)
       MPI_Comm_free(&comm_a);
     if(comm_b != MPI_COMM_NULL)
       MPI_Comm_free(&comm_b);
     MPI_Group_free(&group_a);
     MPI_Group_free(&group_b);
     MPI_Group_free(&MPI_GROUP_WORLD);
     MPI_Finalize();
   }

The library:

   void lib_call(MPI_Comm comm)
   {
     int me, done = 0;
     MPI_Comm_rank(comm, &me);
     if(me == 0)
        while(!done)
        {
           MPI_Recv(..., MPI_ANY_SOURCE, MPI_ANY_TAG, comm);
           ...
        }
     else
     {
       /* work */
       MPI_Send(..., 0, ARBITRARY_TAG, comm);
       ....
     }
#ifdef EXAMPLE_2C
     /* include (resp, exclude) for safety (resp, no safety): */
     MPI_Barrier(comm);
#endif
   }

The above example is really three examples, depending on whether or not one includes rank 3 in list b, and whether or not a synchronize is included in lib call. This example illustrates that, despite contexts, subsequent calls to lib call with the same context need not be safe from one another (colloquially, “back-masking”). Safety is realized if the MPI Barrier is added. What this demonstrates is that libraries have to be written carefully, even with contexts. When rank 3 is excluded, then the synchronize is not needed to get safety from back masking.

Algorithms like “reduce” and “allreduce” have strong enough source selectivity properties so that they are inherently okay (no backmasking), provided that MPI provides basic guarantees. So are multiple calls to a typical tree-broadcast algorithm with the same root or different roots (see [6]). Here we rely on two guarantees of MPI: pairwise ordering of messages between processes in the same context, and source selectivity — deleting either feature removes the guarantee that backmasking cannot be required.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Algorithms that try to do non-deterministic broadcasts or other calls that include wildcard operations will not generally have the good properties of the deterministic implementations of “reduce,” “allreduce,” and “broadcast.” Such algorithms would have to utilize the monotonically increasing tags (within a communicator scope) to keep things straight.

All of the foregoing is a supposition of “collective calls” implemented with point-to-point operations. MPI implementations may or may not implement collective calls using point-to-point operations. These algorithms are used to illustrate the issues of correctness and safety, independent of how MPI implements its collective calls. See also section 1.8.

1.6 Inter-Communication

This section introduces the concept of inter-communication and describes the portions of MPI that support it. It describes support for writing programs that contain user-level servers.

All point-to-point communication described thus far has involved communication between processes that are members of the same group. This type of communication is called “intra-communication” and the communicator used is called an “intra-communicator,” as we have noted earlier in the chapter.

In modular and multi-disciplinary applications, different process groups execute distinct modules and processes within different modules communicate with one another in a pipeline or a more general module graph. In these applications, the most natural way for a process to specify a target process is by the rank of the target process within the target group. In applications that contain internal user-level servers, each server may be a process group that provides services to one or more clients, and each client may be a process group that uses the services of one or more servers. It is again most natural to specify the target process by rank within the target group in these applications. This type of communication is called “inter-communication” and the communicator used is called an “inter-communicator,” as introduced earlier.

An inter-communication is a point-to-point communication between processes in different groups. The group containing a process that initiates an inter-communication operation is called the “local group,” that is, the sender in a send and the receiver in a receive. The group containing the target process is called the “remote group,” that is, the receiver in a send and the sender in a receive. As in intra-communication, the target process is specified using a (communicator, rank) pair. Unlike intra-communication, the rank is relative to a second, remote group.

All inter-communicator constructors are blocking and require that the local and remote groups be disjoint in order to avoid deadlock.

Here is a summary of the properties of inter-communication and inter-communicators:

• The syntax of point-to-point communication is the same for both inter- and intracommunication. The same communicator can be used both for send and for receive operations.

• A target process is addressed by its rank in the remote group, both for sends and for receives.

• Communications using an inter-communicator are guaranteed not to conflict with any communications that use a different communicator.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

• An inter-communicator cannot be used for collective communication.

• A communicator will provide either intra- or inter-communication, never both.

The routine MPI COMM TEST INTER may be used to determine if a communicator is an inter- or intra-communicator. Inter-communicators can be used as arguments to some of the other communicator access routines. Inter-communicators cannot be used as input to some of the constructor routines for intra-communicators (for instance, MPI COMM CREATE).

Advice to implementors. For the purpose of point-to-point communication, communicators can be represented in each process by a tuple consisting of:

group

send context

receive context

source

For inter-communicators, group describes the remote group, and source is the rank of the process in the local group. For intra-communicators, group is the communicator group (remote=local), source is the rank of the process in this group, and send context and receive context are identical. A group is represented by a rank-to-absolute-address translation table.

The inter-communicator cannot be discussed sensibly without considering processes in both the local and remote groups. Imagine a process P in group P, which has an intercommunicator C_{P}, and a process Q in group Q, which has an inter-communicator C_{Q}. Then

• C_{P}.group describes the group Q and C_{Q}.group describes the group P.

• C_{P}.send context = C_{Q}.receive context and the context is unique in Q; C_{P}.receive context = C_{Q}.send context and this context is unique in P.

• C_{P}.source is rank of P in P and C_{Q}.source is rank of Q in Q.

Assume that P sends a message to Q using the inter-communicator. Then P uses the group table to find the absolute address of Q; source and send context are appended to the message.

Assume that Q posts a receive with an explicit source argument using the intercommunicator. Then Q matches receive context to the message context and source argument to the message source.

The same algorithm is appropriate for intra-communicators as well.

In order to support inter-communicator accessors and constructors, it is necessary to supplement this model with additional structures, that store information about the local communication group, and additional safe contexts. (End of advice to implementors.)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

1.6.1 Inter-communicator Accessors

MPI COMM TEST INTER(comm, flag)

IN  comm  communicator (handle)
OUT  flag  (logical)

int MPI Comm test inter(MPI Comm comm, int *flag)
MPI COMM TEST INTER(COMM, FLAG, IERROR)
    INTEGER COMM, IERROR
    LOGICAL FLAG

This local routine allows the calling process to determine if a communicator is an intercommunicator or an intra-communicator. It returns true if it is an inter-communicator, otherwise false.

When an inter-communicator is used as an input argument to the communicator accessors described above under intra-communication, the following table describes behavior.

MPI COMM * Function Behavior (in Inter-Communication Mode)

MPI COMM SIZE returns the size of the local group. MPI COMM GROUP returns the local group. MPI COMM RANK returns the rank in the local group

Furthermore, the operation MPI COMM COMPARE is valid for inter-communicators. Both communicators must be either intra- or inter-communicators, or else MPI UNEQUAL results. Both corresponding local and remote groups must compare correctly to get the results MPI CONGRUENT and MPI SIMILAR. In particular, it is possible for MPI SIMILAR to result because either the local or remote groups were similar but not identical.

The following accessors provide consistent access to the remote group of an intercommunicator:

The following are all local operations.

MPI COMM REMOTE SIZE(comm, size)

IN  comm  inter-communicator (handle)
OUT  size  number of processes in the remote group of comm

(integer)

int MPI Comm remote size(MPI Comm comm, int *size)
MPI COMM REMOTE SIZE(COMM, SIZE, IERROR)
    INTEGER COMM, SIZE, IERROR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI COMM REMOTE GROUP(comm, group)

IN  comm  inter-communicator (handle)
OUT  group  remote group corresponding to comm (handle)

int MPI Comm remote group(MPI Comm comm, MPI Group *group)
MPI COMM REMOTE GROUP(COMM, GROUP, IERROR)
    INTEGER COMM, GROUP, IERROR

Rationale. Symmetric access to both the local and remote groups of an intercommunicator is important, so this function, as well as MPI COMM REMOTE SIZE have been provided. (End of rationale.)

1.6.2 Inter-communicator Operations

This section introduces four blocking inter-communicator operations. MPI INTERCOMM-CREATE is used to bind two intra-communicators into an inter-communicator; the function MPI INTERCOMM MERGE creates an intra-communicator by merging the local and remote groups of an inter-communicator. The functionsMPI COMM DUP andMPI COMM FREE, introduced previously, duplicate and free an inter-communicator, respectively.

Overlap of local and remote groups that are bound into an inter-communicator is prohibited. If there is overlap, then the program is erroneous and is likely to deadlock. (If a process is multithreaded, and MPI calls block only a thread, rather than a process, then “dual membership” can be supported. It is then the user’s responsibility to make sure that calls on behalf of the two “roles” of a process are executed by two independent threads.)

The function MPI INTERCOMM CREATE can be used to create an inter-communicator from two existing intra-communicators, in the following situation: At least one selected member from each group (the “group leader”) has the ability to communicate with the selected member from the other group; that is, a “peer” communicator exists to which both leaders belong, and each leader knows the rank of the other leader in this peer communicator (the two leaders could be the same process). Furthermore, members of each group know the rank of their leader.

Construction of an inter-communicator from two intra-communicators requires separate collective operations in the local group and in the remote group, as well as a point-to-point communication between a process in the local group and a process in the remote group.

In standard MPI implementations (with static process allocation at initialization), the MPI COMM WORLD communicator (or preferably a dedicated duplicate thereof) can be this peer communicator. In dynamic MPI implementations, where, for example, a process may spawn new child processes during an MPI execution, the parent process may be the “bridge” between the old communication universe and the new communication world that includes the parent and its children.

The application topology functions described in chapter ?? do not apply to intercommunicators. Users that require this capability should utilize MPI INTERCOMM MERGE to build an intra-communicator, then apply the graph or cartesian topology capabilities to that intra-communicator, creating an appropriate topology-oriented intra-communicator. Alternatively, it may be reasonable to devise one’s own application topology mechanisms for this case, without loss of generality. ⊤

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI INTERCOMM CREATE(local comm, local leader, peer comm, remote leader, tag, newintercomm)

IN  local comm  local intra-communicator (handle)
IN  local leader  rank of local group leader in local comm (integer)
IN  peer comm  “peer” communicator; significant only at the local leader

(handle)

IN  remote leader  rank of remote group leader in peer comm; significant

only at the local leader (integer)

IN  tag  “safe” tag (integer)
OUT  newintercomm  new inter-communicator (handle)

⊥

int MPI Intercomm create(MPI Comm local comm, int local leader,
             MPI Comm peer comm, int remote leader, int tag,
             MPI Comm *newintercomm)
MPI INTERCOMM CREATE(LOCAL COMM, LOCAL LEADER, PEER COMM, REMOTE LEADER, TAG,
             NEWINTERCOMM, IERROR)
    INTEGER LOCAL COMM, LOCAL LEADER, PEER COMM, REMOTE LEADER, TAG,
    NEWINTERCOMM, IERROR

This call creates an inter-communicator. It is collective over the union of the local and remote groups. Processes should provide identical local comm and local leader arguments within each group. Wildcards are not permitted for remote leader, local leader, and tag.

This call uses point-to-point communication with communicator peer comm, and with tag tag between the leaders. Thus, care must be taken that there be no pending communication on peer comm that could interfere with this communication.

Advice to users. We recommend using a dedicated peer communicator, such as a duplicate of MPI COMM WORLD, to avoid trouble with peer communicators. (End of advice to users.)

MPI INTERCOMM MERGE(intercomm, high, newintracomm)

IN  intercomm  Inter-Communicator (handle)
IN  high  (logical)
OUT  newintracomm  new intra-communicator (handle)

int MPI Intercomm merge(MPI Comm intercomm, int high,
             MPI Comm *newintracomm)
MPI INTERCOMM MERGE(INTERCOMM, HIGH, INTRACOMM, IERROR)
    INTEGER INTERCOMM, INTRACOMM, IERROR
    LOGICAL HIGH

This function creates an intra-communicator from the union of the two groups that are associated with intercomm. All processes should provide the same high value within each

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Figure 1.1: Three-group pipeline.

of the two groups. If processes in one group provided the value high = false and processes in the other group provided the value high = true then the union orders the “low” group before the “high” group. If all processes provided the same high argument then the order of the union is arbitrary. This call is blocking and collective within the union of the two groups.

Advice to implementors. The implementation of MPI INTERCOMM MERGE, MPI COMM FREE and MPI COMM DUP are similar to the implementation of MPI INTERCOMM CREATE, except that contexts private to the input inter-commun-icator are used for communication between group leaders rather than contexts inside a bridge communicator. (End of advice to implementors.)

1.6.3 Inter-Communication Examples

Example 1: Three-Group “Pipeline”

Groups 0 and 1 communicate. Groups 1 and 2 communicate. Therefore, group 0 requires one inter-communicator, group 1 requires two inter-communicators, and group 2 requires 1 inter-communicator.

main(int argc, char **argv)
{
MPI_Comm   myComm;       /* intra-communicator of local sub-group */
MPI_Comm   myFirstComm;  /* inter-communicator */
MPI_Comm   mySecondComm; /* second inter-communicator (group 1 only) */
int membershipKey;
int rank;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
/* User code must generate membershipKey in the range [0, 1, 2] */
membershipKey = rank % 3;
/* Build intra-communicator for local sub-group */
MPI_Comm_split(MPI_COMM_WORLD, membershipKey, rank, &myComm);
/* Build inter-communicators.  Tags are hard-coded. */
if (membershipKey == 0)
{                     /* Group 0 communicates with group 1. */
  MPI_Intercomm_create( myComm, 0, MPI_COMM_WORLD, 1,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Figure 1.2: Three-group ring.

                       1, &myFirstComm);
}
else if (membershipKey == 1)
{              /* Group 1 communicates with groups 0 and 2. */
  MPI_Intercomm_create( myComm, 0, MPI_COMM_WORLD, 0,
                       1, &myFirstComm);
  MPI_Intercomm_create( myComm, 0, MPI_COMM_WORLD, 2,
                       12, &mySecondComm);
}
else if (membershipKey == 2)
{                     /* Group 2 communicates with group 1. */
  MPI_Intercomm_create( myComm, 0, MPI_COMM_WORLD, 1,
                       12, &myFirstComm);
}
/* Do work ... */
switch(membershipKey)  /* free communicators appropriately */
{
case 1:
   MPI_Comm_free(&mySecondComm);
case 0:
case 2:
   MPI_Comm_free(&myFirstComm);
   break;
}
MPI_Finalize();
}

Example 2: Three-Group “Ring”

Groups 0 and 1 communicate. Groups 1 and 2 communicate. Groups 0 and 2 communicate. Therefore, each requires two inter-communicators.

main(int argc, char **argv)
{
MPI_Comm   myComm;      /* intra-communicator of local sub-group */
MPI_Comm   myFirstComm; /* inter-communicators */

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI_Comm   mySecondComm;
MPI_Status status;
int membershipKey;
int rank;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
...
/* User code must generate membershipKey in the range [0, 1, 2] */
membershipKey = rank % 3;
/* Build intra-communicator for local sub-group */
MPI_Comm_split(MPI_COMM_WORLD, membershipKey, rank, &myComm);
/* Build inter-communicators.  Tags are hard-coded. */
if (membershipKey == 0)
{             /* Group 0 communicates with groups 1 and 2. */
  MPI_Intercomm_create( myComm, 0, MPI_COMM_WORLD, 1,
                       1, &myFirstComm);
  MPI_Intercomm_create( myComm, 0, MPI_COMM_WORLD, 2,
                       2, &mySecondComm);
}
else if (membershipKey == 1)
{         /* Group 1 communicates with groups 0 and 2. */
  MPI_Intercomm_create( myComm, 0, MPI_COMM_WORLD, 0,
                       1, &myFirstComm);
  MPI_Intercomm_create( myComm, 0, MPI_COMM_WORLD, 2,
                       12, &mySecondComm);
}
else if (membershipKey == 2)
{        /* Group 2 communicates with groups 0 and 1. */
  MPI_Intercomm_create( myComm, 0, MPI_COMM_WORLD, 0,
                       2, &myFirstComm);
  MPI_Intercomm_create( myComm, 0, MPI_COMM_WORLD, 1,
                       12, &mySecondComm);
}
/* Do some work ... */
/* Then free communicators before terminating... */
MPI_Comm_free(&myFirstComm);
MPI_Comm_free(&mySecondComm);
MPI_Comm_free(&myComm);
MPI_Finalize();
}

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Example 3: Building Name Service for Intercommunication

The following procedures exemplify the process by which a user could create name service for building intercommunicators via a rendezvous involving a server communicator, and a tag name selected by both groups.

After all MPI processes execute MPI INIT, every process calls the example function, Init server(), defined below. Then, if the new world returned is NULL, the process getting NULL is required to implement a server function, in a reactive loop, Do server(). Everyone else just does their prescribed computation, using new world as the new effective “global” communicator. One designated process calls Undo Server() to get rid of the server when it is not needed any longer.

Features of this approach include:

• Support for multiple name servers

• Ability to scope the name servers to specific processes

• Ability to make such servers come and go as desired.

#define INIT_SERVER_TAG_1 666
#define UNDO_SERVER_TAG_1    777
static int server_key_val;
/* for attribute management for server_comm,  copy callback: */
void handle_copy_fn(MPI_Comm *oldcomm, int *keyval, void *extra_state,
void *attribute_val_in, void **attribute_val_out, int *flag)
{
   /* copy the handle */
   *attribute_val_out = attribute_val_in;
   *flag = 1; /* indicate that copy to happen */
}
int Init_server(peer_comm, rank_of_server, server_comm, new_world)
MPI_Comm peer_comm;
int rank_of_server;
MPI_Comm *server_comm;
MPI_Comm *new_world;    /* new effective world, sans server */
{
    MPI_Comm temp_comm, lone_comm;
    MPI_Group peer_group, temp_group;
    int rank_in_peer_comm, size, color, key = 0;
    int peer_leader, peer_leader_rank_in_temp_comm;
    MPI_Comm_rank(peer_comm, &rank_in_peer_comm);
    MPI_Comm_size(peer_comm, &size);
    if ((size < 2) || (0 > rank_of_server) || (rank_of_server >= size))
        return (MPI_ERR_OTHER);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

/* create two communicators, by splitting peer_comm
into the server process, and everyone else */
peer_leader = (rank_of_server + 1) % size;  /* arbitrary choice */
if ((color = (rank_in_peer_comm == rank_of_server)))
{
MPI_Comm_split(peer_comm, color, key, &lone_comm);
MPI_Intercomm_create(lone_comm, 0, peer_comm, peer_leader,
                   INIT_SERVER_TAG_1, server_comm);
MPI_Comm_free(&lone_comm);
*new_world = MPI_COMM_NULL;
}
else
{
MPI_Comm_Split(peer_comm, color, key, &temp_comm);
MPI_Comm_group(peer_comm, &peer_group);
MPI_Comm_group(temp_comm, &temp_group);
MPI_Group_translate_ranks(peer_group, 1, &peer_leader,
temp_group, &peer_leader_rank_in_temp_comm);
MPI_Intercomm_create(temp_comm, peer_leader_rank_in_temp_comm,
                   peer_comm, rank_of_server,
                   INIT_SERVER_TAG_1, server_comm);
/* attach new_world communication attribute to server_comm: */
/* CRITICAL SECTION FOR MULTITHREADING */
if(server_keyval == MPI_KEYVAL_INVALID)
{
    /* acquire the process-local name for the server keyval */
    MPI_keyval_create(handle_copy_fn, NULL,
                                       &server_keyval, NULL);
}
*new_world = temp_comm;
/* Cache handle of intra-communicator on inter-communicator: */
MPI_Attr_put(server_comm, server_keyval, (void *)(*new_world));
}
return (MPI_SUCCESS);
}

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

The actual server process would commit to running the following code:

int Do_server(server_comm)
MPI_Comm server_comm;
{
void init_queue();
int en_queue(), de_queue(); /* keep triplets of integers
                               for later matching (fns not shown) */
MPI_Comm comm;
MPI_Status status;
int client_tag, client_source;
int client_rank_in_new_world, pairs_rank_in_new_world;
int buffer[10], count = 1;
void *queue;
init_queue(&queue);

for (;;)
{
    MPI_Recv(buffer, count, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG,
             server_comm, &status); /* accept from any client */
    /* determine client: */
    client_tag = status.MPI_TAG;
    client_source = status.MPI_SOURCE;
    client_rank_in_new_world = buffer[0];
    if (client_tag == UNDO_SERVER_TAG_1)       /* client that
                                               terminates server */
    {
        while (de_queue(queue, MPI_ANY_TAG, &pairs_rank_in_new_world,
                        &pairs_rank_in_server))
            ;
        MPI_Intercomm_free(&server_comm);
        break;
    }
    if (de_queue(queue, client_tag, &pairs_rank_in_new_world,
                    &pairs_rank_in_server))
    {
        /* matched pair with same tag, tell them
           about each other! */
        buffer[0] = pairs_rank_in_new_world;
        MPI_Send(buffer, 1, MPI_INT, client_src, client_tag,
                                                 server_comm);

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

        buffer[0] = client_rank_in_new_world;
        MPI_Send(buffer, 1, MPI_INT, pairs_rank_in_server, client_tag,
                 server_comm);
    }
    else
        en_queue(queue, client_tag, client_source,
                                    client_rank_in_new_world);
}
}

A particular process would be responsible for ending the server when it is no longer needed. Its call to Undo server would terminate server function.

int Undo_server(server_comm)     /* example client that ends server */
MPI_Comm *server_comm;
{
int buffer = 0;
MPI_Send(&buffer, 1, MPI_INT, 0, UNDO_SERVER_TAG_1, *server_comm);
MPI_Intercomm_free(server_comm);
}

The following is a blocking name-service for inter-communication, with same semantic restrictions as MPI Intercomm create, but simplified syntax. It uses the functionality just defined to create the name service.

int Intercomm_name_create(local_comm, server_comm, tag, comm)
MPI_Comm local_comm, server_comm;
int tag;
MPI_Comm *comm;
{
int error;
int found;   /* attribute acquisition mgmt for new_world */
             /* comm in server_comm */
void *val;
MPI_Comm new_world;
int buffer[10], rank;
int local_leader = 0;
MPI_Attr_get(server_comm, server_keyval, &val, &found);
new_world = (MPI_Comm)val; /* retrieve cached handle */
MPI_Comm_rank(server_comm, &rank);  /* rank in local group */
if (rank == local_leader)
{

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

        buffer[0] = rank;
        MPI_Send(&buffer, 1, MPI_INT, 0, tag, server_comm);
        MPI_Recv(&buffer, 1, MPI_INT, 0, tag, server_comm);
    }
    error = MPI_Intercomm_create(local_comm, local_leader, new_world,
                                 buffer[0], tag, comm);
    return(error);
}

1.7 Caching

MPI provides a “caching” facility that allows an application to attach arbitrary pieces of information, called attributes, to communicators. More precisely, the caching facility allows a portable library to do the following:

• pass information between calls by associating it with an MPI intra- or inter-commun-icator,

• quickly retrieve that information, and

• be guaranteed that out-of-date information is never retrieved, even if the communicator is freed and its handle subsequently reused by MPI.

The caching capabilities, in some form, are required by built-in MPI routines such as collective communication and application topology. Defining an interface to these capabilities as part of the MPI standard is valuable because it permits routines like collective communication and application topologies to be implemented as portable code, and also because it makes MPI more extensible by allowing user-written routines to use standard MPI calling sequences.

Advice to users. The communicator MPI COMM SELF is a suitable choice for posting process-local attributes, via this attributing-caching mechanism. (End of advice to users.)

1.7.1 Functionality

Attributes are attached to communicators. Attributes are local to the process and specific to the communicator to which they are attached. Attributes are not propagated by MPI from one communicator to another except when the communicator is duplicated using MPI COMM DUP (and even then the application must give specific permission through

⊤ callback functions for the attribute to be copied).

Advice to users. Attributes are in C of type void *. Typically, such attribute will be a pointer to a structure that contains further information, or a handle to an MPI object. In Fortran, attributes are of type INTEGER. Such attribute can be a handle to an MPI object, or just an integer-valued attribute. (End of advice to users.)

⊥

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Advice to implementors. Attributes are scalar values, equal in size to, or larger than a C-language pointer. Attributes can always hold an MPI handle. (End of advice to implementors.)

The caching interface defined here represents that attributes be stored by MPI opaquely within a communicator. Accessor functions include the following:

• obtain a key value (used to identify an attribute); the user specifies “callback” functions by which MPI informs the application when the communicator is destroyed or copied.

• store and retrieve the value of an attribute;

Advice to implementors. Caching and callback functions are only called synchronously, in response to explicit application requests. This avoid problems that result from repeated crossings between user and system space. (This synchronous calling rule is a general property of MPI.) The choice of key values is under control of MPI. This allows MPI to optimize its implementation of attribute sets. It also avoids conflict between independent modules caching information on the same communicators. A much smaller interface, consisting of just a callback facility, would allow the entire caching facility to be implemented by portable code. However, with the minimal callback interface, some form of table searching is implied by the need to handle arbitrary communicators. In contrast, the more complete interface defined here permits rapid access to attributes through the use of pointers in communicators (to find the attribute table) and cleverly chosen key values (to retrieve individual attributes). In light of the efficiency “hit” inherent in the minimal interface, the more complete interface defined here is seen to be superior. (End of advice to implementors.)

MPI provides the following services related to caching. They are all process local.

MPI KEYVAL CREATE(copy fn, delete fn, keyval, extra state)

IN  copy fn  Copy callback function for keyval
IN  delete fn  Delete callback function for keyval
OUT  keyval  key value for future access (integer)
IN  extra state  Extra state for callback functions

int MPI Keyval create(MPI Copy function *copy fn, MPI Delete function
         *delete fn, int *keyval, void* extra state)
MPI KEYVAL CREATE(COPY FN, DELETE FN, KEYVAL, EXTRA STATE, IERROR)
EXTERNAL COPY FN, DELETE FN
INTEGER KEYVAL, EXTRA STATE, IERROR

Generates a new attribute key. Keys are locally unique in a process, and opaque to user, though they are explicitly stored in integers. Once allocated, the key value can be used to associate attributes and access them on any locally defined communicator.

The copy fn function is invoked when a communicator is duplicated by MPI COMM DUP. copy fn should be of type MPI Copy function, which is defined as follows: ⊤

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

typedef int MPI_Copy_function(MPI_Comm oldcomm, int keyval,
                              void *extra_state, void *attribute_val_in,
                              void *attribute_val_out, int *flag)

⊥

A Fortran declaration for such a function is as follows:

⊤

PROCEDURE COPY FUNCTION(OLDCOMM, KEYVAL, EXTRA STATE, ATTRIBUTE VAL IN,
             ATTRIBUTE VAL OUT, FLAG, IERR)
    INTEGER OLDCOMM, KEYVAL, EXTRA STATE, ATTRIBUTE VAL IN,
    ATTRIBUTE VAL OUT, IERR
    LOGICAL FLAG

⊥

The copy callback function is invoked for each key value in oldcomm in arbitrary order. Each call to the copy callback is made with a key value and its corresponding attribute. If it returns flag = 0, then the attribute is deleted in the duplicated communicator. Oth-

⊤ erwise (flag = 1), the new attribute value is set to the value returned in attribute val out. ⊥ The function returns MPI SUCCESS on success and an error code on failure (in which case ⊤ MPI COMM DUP will fail).

copy fn may be specified as MPI NULL COPY FN or MPI DUP FN from either C or FORTRAN; MPI NULL COPY FN is a function that does nothing other than returning flag = 0 and MPI SUCCESS. MPI DUP FN is a simple-minded copy function that sets flag = 1, returns the value of attribute val in in attribute val out, and returns MPI SUCCESS.

Advice to users. Even both formal arguments attribute val in and attribute val out are of type void *, their usage differ. The C copy function is passed by MPI in attribute val in the value of the attribute, and in attribute val out the address of the attribute, so as to allow the function to return the (new) attribute value. The use of

⊥ type void * for both is to avoid messy type casts.

A valid copy function is one that completely duplicates the information by making a full duplicate copy of the data structures implied by an attribute; another might just make another reference to that data structure, while using a reference-count mechanism. Other types of attributes might not copy at all (they might be specific to oldcomm only). (End of advice to users.)

⊤

Advice to implementors. A C interface should be assumed for copy and delete functions associated with key values created in C; a Fortran calling interface should be assumed for key values created in Fortran. (End of advice to implementors.)

⊥

Analogous to copy fn is a callback deletion function, defined as follows. The delete fn function is invoked when a communicator is deleted by MPI COMM FREE or when a call is made explicitly to MPI ATTR DELETE. delete fn should be of type MPI Delete function,

⊤ which is defined as follows:

typedef int MPI_Delete_function(MPI_Comm comm, int keyval,
void *attribute_val, void *extra_state);

A Fortran declaration for such a function is as follows:

PROCEDURE DELETE FUNCTION(COMM, KEYVAL, ATTRIBUTE VAL, EXTRA STATE, IERR)
    INTEGER COMM, KEYVAL, ATTRIBUTE VAL, EXTRA STATE, IERR

⊥

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

This function is called by MPI COMM FREE and MPI ATTR DELETE to do whatever

⊤ is needed to remove an attribute. The function returns MPI SUCCESS on success and an

error code on failure (in which case MPI COMM FREE will fail).

delete fn may be specified as MPI NULL DELETE FN from either C or FORTRAN; MPI NULL DELETE FN is a function that does nothing, other than returning MPI SUCCESS.

⊥

The special key value MPI KEYVAL INVALID is never returned by MPI KEYVAL CREATE. Therefore, it can be used for static initialization of key values.

MPI KEYVAL FREE(keyval)

INOUT keyval Frees the integer key value (integer)

int MPI Keyval free(int *keyval)
MPI KEYVAL FREE(KEYVAL, IERROR)
    INTEGER KEYVAL, IERROR

Frees an extant attribute key. This function sets the value of keyval to MPI KEYVAL INVALID. Note that it is not erroneous to free an attribute key that is in use, because the actual free does not transpire until after all references (in other communicators on the process) to the key have been freed. These references need to be explictly freed by the program, either via calls to MPI ATTR DELETE that free one attribute instance, or by calls to MPI COMM FREE that free all attribute instances associated with the freed communicator. ⊤

⊥

MPI ATTR PUT(comm, keyval, attribute val)

IN  comm  communicator to which attribute will be attached (han-

dle)

IN  keyval  key value, as returned by

MPI KEYVAL CREATE (integer)

IN  attribute val  attribute value

int MPI Attr put(MPI Comm comm, int keyval, void* attribute val)
MPI ATTR PUT(COMM, KEYVAL, ATTRIBUTE VAL, IERROR)
    INTEGER COMM, KEYVAL, ATTRIBUTE VAL, IERROR

This function stores the stipulated attribute value attribute val for subsequent retrieval by MPI ATTR GET. If the value is already present, then the outcome is as if MPI ATTR-DELETE was first called to delete the previous value (and the callback function delete fn was executed), and a new value was next stored. The call is erroneous if there is no key with value keyval; in particular MPI KEYVAL INVALID is an erroneous key value. The call ⊤ will fail if the delete fn function returned an error code other than MPI SUCCESS. ⊥

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI ATTR GET(comm, keyval, attribute val, flag)

IN  comm  communicator to which attribute is attached (handle)
IN  keyval  key value (integer)
OUT  attribute val  attribute value, unless flag = false
OUT  flag  true if an attribute value was extracted; false if no

attribute is associated with the key

⊤

int MPI Attr get(MPI Comm comm, int keyval, void *attribute val, int *flag)

⊥

MPI ATTR GET(COMM, KEYVAL, ATTRIBUTE VAL, FLAG, IERROR)
    INTEGER COMM, KEYVAL, ATTRIBUTE VAL, IERROR
    LOGICAL FLAG

Retrieves attribute value by key. The call is erroneous if there is no key with value keyval. On the other hand, the call is correct if the key value exists, but no attribute is attached on comm for that key; in such case, the call returns flag = false. In particular

⊤ MPI KEYVAL INVALID is an erroneous key value.

Advice to users. The call to MPI Attr put passes in attribute val the value of the attribute; the call to MPI Attr get passes in attribute val the address of the the location where the attribute value is to be returned. Thus, if the attribute value itself is a pointer of type void*, the the actual attribute val parameter to MPI Attr put will be of type void* and the actual attribute val parameter to MPI Attr put will be of type void**. (End of advice to users.)

Rationale. The use of a formal parameter attribute val or type void* (rather than void**) avoids the messy type casting that would be needed if the attribute value is declared with a type other than void*. (End of rationale.)

⊥

MPI ATTR DELETE(comm, keyval)

IN  comm  communicator to which attribute is attached (handle)
IN  keyval  The key value of the deleted attribute (integer)

int MPI Attr delete(MPI Comm comm, int keyval)
MPI ATTR DELETE(COMM, KEYVAL, IERROR)
    INTEGER COMM, KEYVAL, IERROR

Delete attribute from cache by key. This function invokes the attribute delete function

⊤ delete fn specified when the keyval was created. The call will fail if the delete fn function ⊥ returns an error code other than MPI SUCCESS.

Whenever a communicator is replicated using the function MPI COMM DUP, all callback copy functions for attributes that are currently set are invoked (in arbitrary order). Whenever a communicator is deleted using the function MPI COMM FREE all callback delete functions for attributes that are currently set are invoked.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

1.7.2 Attributes Example

Advice to users. This example shows how to write a collective communication operation that uses caching to be more efficient after the first call. The coding style assumes that MPI function results return only error statuses. (End of advice to users.)

/* key for this module's stuff: */
static int gop_key = MPI_KEYVAL_INVALID;
typedef struct
{
 int ref_count;          /* reference count */
 /* other stuff, whatever else we want */
} gop_stuff_type;
Efficient_Collective_Op (comm, ...)
MPI_Comm comm;
{
gop_stuff_type *gop_stuff;
MPI_Group       group;
int             foundflag;
MPI_Comm_group(comm, &group);
if (gop_key == MPI_KEYVAL_INVALID) /* get a key on first call ever */
{
  if ( ! MPI_keyval_create( gop_stuff_copier,
                           gop_stuff_destructor,
                           &gop_key, (void *)0));
  /* get the key while assigning its copy and delete callback
     behavior. */
  MPI_Abort (comm, 99);
}
MPI_Attr_get (comm, gop_key, &gop_stuff, &foundflag);
if (foundflag)
{ /* This module has executed in this group before.
     We will use the cached information */
}
else
{ /* This is a group that we have not yet cached anything in.
     We will now do so.
  */
  /* First, allocate storage for the stuff we want,
     and initialize the reference count */
  gop_stuff = (gop_stuff_type *) malloc (sizeof(gop_stuff_type));

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

    if (gop_stuff == NULL) { /* abort on out-of-memory error */ }
    gop_stuff -> ref_count = 1;
    /* Second, fill in *gop_stuff with whatever we want.
       This part isn't shown here */
    /* Third, store gop_stuff as the attribute value */
    MPI_Attr_put ( comm, gop_key, gop_stuff);
  }
  /* Then, in any case, use contents of *gop_stuff
     to do the global op ... */
}
/* The following routine is called by MPI when a group is freed */
gop_stuff_destructor (comm, keyval, gop_stuff, extra)
MPI_Comm comm;
int keyval;
gop_stuff_type *gop_stuff;
void *extra;
{
  if (keyval != gop_key) { /* abort -- programming error */ }
  /* The group's being freed removes one reference to gop_stuff */
  gop_stuff -> ref_count -= 1;
  /* If no references remain, then free the storage */
  if (gop_stuff -> ref_count == 0) {
    free((void *)gop_stuff);
  }
}
/* The following routine is called by MPI when a group is copied */
gop_stuff_copier (comm, keyval, extra, gop_stuff_in, gop_stuff_out, flag)
MPI_Comm comm;
int keyval;
gop_stuff_type *gop_stuff_in, *gop_stuff_out;
void *extra;
{
  if (keyval != gop_key) { /* abort -- programming error */ }
  /* The new group adds one reference to this gop_stuff */
  gop_stuff -> ref_count += 1;
  gop_stuff_out = gop_stuff_in;
}

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

1.8 Formalizing the Loosely Synchronous Model

In this section, we make further statements about the loosely synchronous model, with particular attention to intra-communication.

1.8.1 Basic Statements

When a caller passes a communicator (that contains a context and group) to a callee, that communicator must be free of side effects throughout execution of the subprogram: there should be no active operations on that communicator that might involve the process. This provides one model in which libraries can be written, and work “safely.” For libraries so designated, the callee has permission to do whatever communication it likes with the communicator, and under the above guarantee knows that no other communications will interfere. Since we permit good implementations to create new communicators without synchronization (such as by preallocated contexts on communicators), this does not impose a significant overhead.

This form of safety is analogous to other common computer-science usages, such as passing a descriptor of an array to a library routine. The library routine has every right to expect such a descriptor to be valid and modifiable.

1.8.2 Models of Execution

In the loosely synchronous model, transfer of control to a parallel procedure is effected by having each executing process invoke the procedure. The invocation is a collective operation: it is executed by all processes in the execution group, and invocations are similarly ordered at all processes. However, the invocation need not be synchronized.

We say that a parallel procedure is active in a process if the process belongs to a group that may collectively execute the procedure, and some member of that group is currently executing the procedure code. If a parallel procedure is active in a process, then this process may be receiving messages pertaining to this procedure, even if it does not currently execute the code of this procedure.

Static communicator allocation

This covers the case where, at any point in time, at most one invocation of a parallel procedure can be active at any process, and the group of executing processes is fixed. For example, all invocations of parallel procedures involve all processes, processes are singlethreaded, and there are no recursive invocations.

In such a case, a communicator can be statically allocated to each procedure. The static allocation can be done in a preamble, as part of initialization code. If the parallel procedures can be organized into libraries, so that only one procedure of each library can be concurrently active in each processor, then it is sufficient to allocate one communicator per library.

Dynamic communicator allocation

Calls of parallel procedures are well-nested if a new parallel procedure is always invoked in a subset of a group executing the same parallel procedure. Thus, processes that execute the same parallel procedure have the same execution stack.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

In such a case, a new communicator needs to be dynamically allocated for each new invocation of a parallel procedure. The allocation is done by the caller. A new communicator can be generated by a call to MPI COMM DUP, if the callee execution group is identical to the caller execution group, or by a call to MPI COMM SPLIT if the caller execution group is split into several subgroups executing distinct parallel routines. The new communicator is passed as an argument to the invoked routine.

The need for generating a new communicator at each invocation can be alleviated or avoided altogether in some cases: If the execution group is not split, then one can allocate a stack of communicators in a preamble, and next manage the stack in a way that mimics the stack of recursive calls.

One can also take advantage of the well-ordering property of communication to avoid confusing caller and callee communication, even if both use the same communicator. To do so, one needs to abide by the following two rules:

• messages sent before a procedure call (or before a return from the procedure) are also received before the matching call (or return) at the receiving end;

• messages are always selected by source (no use is made of MPI ANY SOURCE).

The General case

In the general case, there may be multiple concurrently active invocations of the same parallel procedure within the same group; invocations may not be well-nested. A new communicator needs to be created for each invocation. It is the user’s responsibility to make sure that, should two distinct parallel procedures be invoked concurrently on overlapping sets of processes, then communicator creation be properly coordinated.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Bibliography

[1] Purushotham V. Bangalore, Nathan E. Doss, and Anthony Skjellum. MPI++: Issues and Features. In OON-SKI ’94, page in press, 1994.

[2] D. Feitelson. Communicators: Object-based multiparty interactions for parallel programming. Technical Report 91-12, Dept. Computer Science, The Hebrew University of Jerusalem, November 1991.

[3] A. Skjellum and A. Leung. Zipcode: a portable multicomputer communication library atop the reactive kernel. In D. W. Walker and Q. F. Stout, editors, Proceedings of the Fifth Distributed Memory Concurrent Computing Conference, pages 767–776. IEEE Press, 1990.

[4] Anthony Skjellum, Nathan E. Doss, and Purushotham V. Bangalore. Writing Libraries in MPI. In Anthony Skjellum and Donna S. Reese, editors, Proceedings of the Scalable Parallel Libraries Conference, pages 166–173. IEEE Computer Society Press, October 1993.

[5] Anthony Skjellum, Steven G. Smith, Nathan E. Doss, Alvin P . Leung, and Manfred Morari. The Design and Evolution of Zipcode. Parallel Computing, 1994. (Invited Paper, to appear in Special Issue on Message Passing).

[6] Anthony Skjellum, Steven G. Smith, Nathan E. Doss, Charles H. Still, Alvin P. Leung, and Manfred Morari. Zipcode: A Portable Communication Layer for High Performance Multicomputing. Technical Report UCRL-JC-106725 (revised 9/92, 12/93, 4/94), Lawrence Livermore National Laboratory, March 1991. To appear in Concurrency: Practice & Experience.

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47