D R A F T

Document for a Standard Message-Passing Interface

Message Passing Interface Forum

May 30, 1995

This work was supported in part by ARPA and NSF under grant ASC-9310330, the

National Science Foundation Science and Technology Center Cooperative Agreement No. CCR-8809615, and by the Commission of the European Community

through Esprit project P6643.

This is the result of a LaTeX run of a draft of a single chapter of the MPIF Final Report document.

Chapter 1

MPI Environmental Management

This chapter discusses routines for getting and, where appropriate, setting various parameters that relate to the MPI implementation and the execution environment (such as error handling). The procedures for entering and leaving the MPI execution environment are also described here.

1.1 Implementation information

1.1.1 Environmental Inquiries

A set of attributes that describe the execution environment are attached to the communicator MPI COMM WORLD when MPI is initialized. The value of these attributes can be inquired by using the function MPI ATTR GET described in Chapter ??. It is erroneous to delete these attributes, free their keys, or change their values. ⊤

The list of predefined attribute keys include ⊥

MPI TAG UB Upper bound for tag value.

MPI HOST Host process rank, if such exists, MPI PROC NULL, otherwise.

MPI IO rank of a node that has regular I/O facilities (possibly myrank). Nodes in the same communicator may return different values for this parameter. ⊤

MPI WTIME IS GLOBAL Boolean variable that indicates whether clocks are synchronized. ⊥

Vendors may add implementation specific parameters (such as node number, real memory size, virtual memory size, etc.) ⊤

These predefined attributes do not change value between MPI initialization (MPI INIT and MPI completion (MPI FINALIZE), and cannot be updated or deleted by users.

Advice to users. Note that in the C binding, the value returned by these attributes is a pointer to an int containing the requested value. (End of advice to users.)

⊥

The required parameter values are discussed in more detail below:

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

Tag values

Tag values range from 0 to the value returned for MPI TAG UB inclusive. These values are guaranteed to be unchanging during the execution of an MPI program. In addition, the tag upper bound value must be at least 32767. An MPI implementation is free to make the value of MPI TAG UB larger than this; for example, the value 2^{30} − 1 is also a legal value for

⊤ MPI TAG UB.

⊥ The attribute MPI TAG UB has the same value on all processes of MPI COMM WORLD.

Host rank

The value returned for MPI HOST gets the rank of the HOST process in the group associated with communicator MPI COMM WORLD, if there is such. MPI PROC NULL is returned if there is no host. MPI does not specify what it means for a process to be a HOST, nor does

⊤ it requires that a HOST exists. ⊥ The attribute MPI HOST has the same value on all processes of MPI COMM WORLD.

IO rank

The value returned for MPI IO is the rank of a processor that can provide language-standard I/O facilities. For Fortran, this means that all of the Fortran I/O operations are supported (e.g., OPEN, REWIND, WRITE). For C, this means that all of the ANSI-C I/O operations are

⊤ supported (e.g., fopen, fprintf, lseek).

If every process can provide language-standard I/O, then the value MPI ANY SOURCE will be returned. Otherwise, if the calling process can provide language-standard I/O, then its rank will be returned. Otherwise, if some process can provide language-standard I/O then the rank of one such process will be returned. The same value need not be returned by all processes. If no process can provide language-standard I/O, then the value MPI PROC NULL will be returned.

Advice to users. Note that input is not collective, and this attribute does not indicate which process can or does provide input. (End of advice to users.)

Clock synchronization

The value returned for MPI WTIME IS GLOBAL is 1 if clocks at all processes in MPI COMM WORLD are synchronized, 0 otherwise. A collection of clocks is considered synchronized if explicit effort has been taken to synchronize them. The expectation is that the variation in time, as measured by calls to MPI WTIME, will be less then one half the round-trip time for an MPI message of length zero. If time is measured at a process just before a send and at another process just after a matching receive, the second time should be always higher than the first one.

The attribute MPI WTIME IS GLOBAL need not be present when the clocks are not synchronized (however, the attribute key MPI WTIME IS GLOBAL is always valid). This attribute may be associated with communicators other then MPI COMM WORLD.

The attribute MPI WTIME IS GLOBAL has the same value on all processes of MPI COMM WORLD.

⊥

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI GET PROCESSOR NAME( name, resultlen )

OUT  name  A unique specifier for the actual (as opposed to vir-

tual) node.

OUT  resultlen  Length (in printable characters) of the result returned

                                      in name
int MPI Get processor name(char *name, int *resultlen)
MPI GET PROCESSOR NAME( NAME, RESULTLEN, IERROR)
    CHARACTER*(*) NAME
    INTEGER RESULTLEN,IERROR

This routine returns the name of the processor on which it was called at the moment of the call. The name is a character string for maximum flexibility. From this value it must be possible to identify a specific piece of hardware; possible values include “processor 9 in rack 4 of mpp.cs.org” and “231” (where 231 is the actual processor number in the running homogeneous system). The argument name must represent storage that is at least MPI MAX PROCESSOR NAME characters long. MPI GET PROCESSOR NAME may write up to this many characters into name.

The number of characters actually written is returned in the output argument, resultlen.

Rationale. This function allows MPI implementations that do process migration to return the current processor. Note that nothing in MPI requires or defines process migration; this definition of MPI GET PROCESSOR NAME simply allows such an implementation. (End of rationale.)

Advice to users. The user must provide at least MPI MAX PROCESSOR NAME space to write the processor name — processor names can be this long. The user should examine the ouput argument, resultlen, to determine the actual length of the name. (End of advice to users.)

⊤

The constant MPI BSEND OVERHEAD provides an upper bound on the fixed overhead per message buffered by a call to MPI BSEND (see Section ??). ⊥

1.2 Error handling

An MPI implementation cannot or may choose not to handle some errors that occur during MPI calls. These can include errors that generate exceptions or traps, such as floating point errors or access violations. The set of errors that are handled by MPI is implementationdependent. Each such error generates an MPI exception. ⊤

The above text takes precendence over any text on error handling within this document. Specifically, text that states that errors will be handled should be read as may be handled. ⊥

A user can associate an error handler with a communicator. The specified error handling routine will be used for any MPI exception that occurs during a call to MPI for a communication with this communicator. MPI calls that are not related to any communicator are considered to be attached to the communicator MPI COMM WORLD. The attachment of error handlers to communicators is purely local: different processes may attach different error handlers to the same communicator.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

A newly created communicator inherits the error handler that is associated with the “parent” communicator. In particular, the user can specify a “global” error handler for all communicators by associating this handler with the communicator MPI COMM WORLD immediately after initialization.

Several predefined error handlers are available in MPI:

MPI ERRORS ARE FATAL The handler, when called, causes the program to abort on all executing processes. This has the same effect as if MPI ABORT was called by the process that invoked the handler.

⊤ MPI ERRORS RETURN The handler has no effect other than returning the error code to the ⊥ user.

Implementations may provide additional predefined error handlers and programmers can code their own error handlers.

The error handler MPI ERRORS ARE FATAL is associated by default with MPI COMM-WORLD after initialization. Thus, if the user chooses not to control error handling, every error that MPI handles is treated as fatal. Since (almost) all MPI calls return an error code, a user may choose to handle errors in its main code, by testing the return code of MPI calls and executing a suitable recovery code when the call was not successful. In this case, the error handler MPI ERRORS RETURN will be used. Usually it is more convenient and more efficient not to test for errors after each MPI call, and have such error handled by a non trivial MPI error handler.

After an error is detected, the state of MPI is undefined. That is, using a user-defined error handler, or MPI ERRORS RETURN, does not necessarily allow the user to continue to use MPI after an error is detected. The purpose of these error handlers is to allow a user to issue user-defined error messages and to take actions unrelated to MPI (such as flushing I/O buffers) before a program exits. An MPI implementation is free to allow MPI to continue after an error but is not required to do so.

Advice to implementors. A good quality implementation will, to the greatest possible, extent, circumscribe the impact of an error, so that normal processing can continue after an error handler was invoked. The implementation documentation will provide information on the possible effect of each class of errors. (End of advice to implementors.)

An MPI error handler is an opaque object, which is accessed by a handle. MPI calls are provided to create new error handlers, to associate error handlers with communicators, and to test which error handler is associated with a communicator.

MPI ERRHANDLER CREATE( function, errhandler )

IN  function  user defined error handling procedure
OUT  errhandler  MPI error handler (handle)

int MPI Errhandler create(MPI Handler function *function,
             MPI Errhandler *errhandler)
MPI ERRHANDLER CREATE(FUNCTION, HANDLER, IERROR)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

    EXTERNAL FUNCTION
    INTEGER ERRHANDLER, IERROR

Register the user routine function for use as an MPI exception handler. Returns in errhandler a handle to the registered exception handler. ⊤

In the C language, the user routine should be a C function of type MPI Handler function, ⊥ which is defined as:

typedef void (MPI_Handler_function)(MPI_Comm *, int *, ...);

The first argument is the communicator in use. The second is the error code to be re- ⊤ turned by the MPI routine that raised the error. If the routine would have returned MPI ERR IN STATUS, it is the error code returned in the status for the request that caused the error handler to be invoked. The remaining arguments are “stdargs” arguments whose ⊥ number and meaning is implementation-dependent. An implementation should clearly document these arguments. Addresses are used so that the handler may be written in Fortran.

Rationale. The variable argument list is provided because it provides an ANSIstandard hook for providing additional information to the error handler; without this hook, ANSI C prohibits additional arguments. (End of rationale.)

MPI ERRHANDLER SET( comm, errhandler )

IN  comm  communicator to set the error handler for (handle)
IN  errhandler  new MPI error handler for communicator (handle)

int MPI Errhandler set(MPI Comm comm, MPI Errhandler errhandler)
MPI ERRHANDLER SET(COMM, ERRHANDLER, IERROR)
    INTEGER COMM, ERRHANDLER, IERROR

Associates the new error handler errorhandler with communicator comm at the calling process. Note that an error handler is always associated with the communicator.

MPI ERRHANDLER GET( comm, errhandler )

IN  comm  communicator to get the error handler from (handle)
OUT  errhandler  MPI error handler currently associated with commu-

nicator (handle)

int MPI Errhandler get(MPI Comm comm, MPI Errhandler *errhandler)
MPI ERRHANDLER GET(COMM, ERRHANDLER, IERROR)
    INTEGER COMM, ERRHANDLER, IERROR

Returns in errhandler (a handle to) the error handler that is currently associated with communicator comm.

Example: A library function may register at its entry point the current error handler for a communicator, set its own private error handler for this communicator, and restore before exiting the previous error handler.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

MPI ERRHANDLER FREE( errhandler )

IN  errhandler  MPI error handler (handle)

int MPI Errhandler free(MPI Errhandler *errhandler)
MPI ERRHANDLER FREE(ERRHANDLER, IERROR)
    INTEGER ERRHANDLER, IERROR

Marks the error handler associated with errhandler for deallocation and sets errhandler to MPI ERRHANDLER NULL. The error handler will be deallocated after all communicators associated with it have been deallocated.

MPI ERROR STRING( errorcode, string, resultlen )

IN  errorcode  Error code returned by an MPI routine
OUT  string  Text that corresponds to the errorcode
OUT  resultlen  Length (in printable characters) of the result returned

in string

int MPI Error string(int errorcode, char *string, int *resultlen)
MPI ERROR STRING(ERRORCODE, STRING, RESULTLEN, IERROR)
    INTEGER ERRORCODE, RESULTLEN, IERROR
    CHARACTER*(*) STRING

⊤ Returns the error string associated with an error code or class. The argument string ⊥ must represent storage that is at least MPI MAX ERROR STRING characters long.

The number of characters actually written is returned in the output argument, resultlen.

Rationale. The form of this function was chosen to make the Fortran and C bindings similar. A version that returns a pointer to a string has two difficulties. First, the return string must be statically allocated and different for each error message (allowing the pointers returned by successive calls to MPI ERROR STRING to point to the correct message). Second, in Fortran, a function declared as returning CHARACTER*(*) can not be referenced in, for example, a PRINT statement. (End of rationale.)

1.3 Error codes and classes

The error codes returned by MPI are left entirely to the implementation (with the exception of MPI SUCCESS). This is done to allow an implementation to provide as much information as possible in the error code (for use with MPI ERROR STRING).

To make it possible for an application to interpret an error code, the routine MPI ERR-

⊤ OR CLASS converts any error code into one of a small set of standard error codes, called error classes. Valid error classes include

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

MPI SUCCESS  No error
MPI ERR BUFFER  Invalid buffer pointer
MPI ERR COUNT  Invalid count argument
MPI ERR TYPE  Invalid datatype argument
MPI ERR TAG  Invalid tag argument
MPI ERR COMM  Invalid communicator
MPI ERR RANK  Invalid rank
MPI ERR REQUEST Invalid request (handle)
MPI ERR ROOT  Invalid root
MPI ERR GROUP  Invalid group
MPI ERR OP  Invalid operation
MPI ERR TOPOLOGY Invalid topology
MPI ERR DIMS  Invalid dimension argument
MPI ERR ARG  Invalid argument of some other kind
MPI ERR UNKNOWN Unknown error
MPI ERR TRUNCATE Message truncated on receive
MPI ERR OTHER  Known error not in this list
MPI ERR INTERN  Internal MPI (implementation) error
MPI ERR IN STATUS Error code is in status
MPI ERR PENDING  Pending request
MPI ERR LASTCODE Last error code

The error classes are a subset of the error codes: an MPI function may return an error class number; and the function MPI ERROR STRING can be used to compute the error string associated with an error class.

The error codes satisfy,

0 = MPI SUCCESS < MPI ERR ... ≤ MPI ERR LASTCODE.

⊥

Rationale. The difference between MPI ERR UNKNOWN and MPI ERR OTHER is that MPI ERROR STRING can return useful information about MPI ERR OTHER.

Note that MPI SUCCESS = 0 is necessary to be consistent with C practice; the separation of error classes and error codes allows us to define the error classes this way. Having a known LASTCODE is often a nice sanity check as well. (End of rationale.)

MPI ERROR CLASS( errorcode, errorclass )

IN  errorcode  Error code returned by an MPI routine
OUT  errorclass  Error class associated with errorcode

int MPI Error class(int errorcode, int *errorclass)
MPI ERROR CLASS(ERRORCODE, ERRORCLASS, IERROR)
INTEGER ERRORCODE, ERRORCLASS, IERROR

⊤

The function MPI ERROR CLASS maps each standard error code (error class) onto itself. ⊥

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

1.4 Timers and synchronization

MPI defines a timer. A timer is specified even though it is not “message-passing,” because timing parallel programs is important in “performance debugging” and because existing timers (both in POSIX 1003.1-1988 and 1003.4D 14.1 and in Fortran 90) are either inconvenient or do not provide adequate access to high-resolution timers.

MPI WTIME()

double MPI Wtime(void)
DOUBLE PRECISION MPI WTIME()

MPI WTIME returns a floating-point number of seconds, representing elapsed wall-clock time since some time in the past.

The “time in the past” is guaranteed not to change during the life of the process. The user is responsible for converting large numbers of seconds to other units if they are preferred.

This function is portable (it returns seconds, not “ticks”), it allows high-resolution,

⊤ and carries no unnecessary baggage. One would use it like this:

{
   double starttime, endtime;
   starttime = MPI_Wtime();
    ....  stuff to be timed  ...
   endtime   = MPI_Wtime();
   printf("That took %f seconds\n",endtime-starttime);
}

⊥

The times returned are local to the node that called them. There is no requirement that different nodes return “the same time.”

MPI WTICK()

double MPI Wtick(void)
DOUBLE PRECISION MPI WTICK()

MPI WTICK returns the resolution of MPI WTIME in seconds. That is, it returns, as a double precision value, the number of seconds between successive clock ticks. For example, if the clock is implemented by the hardware as a counter that is incremented every millisecond, the value returned by MPI WTICK should be 10^{−3}.

1.5 Startup

One goal of MPI is to achieve source code portability. By this we mean that a program written using MPI and complying with the relevant language standards is portable as written, and must not require any source code changes when moved from one system to another. This explicitly does not say anything about how an MPI program is started or launched from

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

the command line, nor what the user must do to set up the environment in which an MPI program will run. However, an implementation may require some setup to be performed before other MPI routines may be called. To provide for this, MPI includes an initialization routine MPI INIT.

MPI INIT()

int MPI Init(int *argc, char ***argv)
MPI INIT(IERROR)
    INTEGER IERROR

This routine must be called before any other MPI routine. It must be called at most once; subsequent calls are erroneous (see MPI INITIALIZED).

All MPI programs must contain a call to MPI init; this routine must be called before any other MPI routine (apart from MPI INITIALIZED) is called. The version for ANSI C accepts the argc and argv that are provided by the arguments to main:

MPI_init( argc, argv );

The Fortran version takes only IERROR. ⊤

An MPI implementation is free to require that the arguments in the C binding must be the arguments to main.

Rationale. The command line arguements are provided to MPI Init to allow an MPI implementation to use them in initializing the MPI environment. They are passed by reference to allow an MPI implementation to provide them in environments where the command-line arguments are not provided to main. (End of rationale.)

⊥

MPI FINALIZE()

int MPI Finalize(void)
MPI FINALIZE(IERROR)
    INTEGER IERROR

This routines cleans up all MPI state. Once this routine is called, no MPI routine (even MPI INIT) may be called. The user must ensure that all pending communications involving a process completes before the process calls MPI FINALIZE.

MPI INITIALIZED( flag )

OUT  flag  Flag is true if MPI INIT has been called and false

otherwise.

int MPI Initialized(int *flag)
MPI INITIALIZED(FLAG, IERROR)
    LOGICAL FLAG

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

    INTEGER IERROR

This routine may be used to determine whether MPI INIT has been called. It is the only routine that may be called before MPI INIT is called.

MPI ABORT( comm, errorcode )

IN  comm  communicator of tasks to abort
IN  errorcode  error code to return to invoking environment

int MPI Abort(MPI Comm comm, int errorcode)
MPI ABORT(COMM, ERRORCODE, IERROR)
    INTEGER COMM, ERRORCODE, IERROR

This routine makes a “best attempt” to abort all tasks in the group of comm. This function does not require that the invoking environment take any action with the error code. However, a Unix or POSIX environment should handle this as a return errorcode from the main program or an abort(errorcode).

MPI implementations are required to define the behavior of MPI ABORT at least for a comm of MPI COMM WORLD. MPI implementations may ignore the comm argument and act as if the comm was MPI COMM WORLD.

Rationale. The communicator argument is provided to allow for future extensions of MPI to environments with, for example, dynamic process management. In particular, it allows but does not require an MPI implementation to abort a subset of MPI COMM WORLD. (End of rationale.)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47