Improve ABI for identifying an MPI implementation

[simonbyrne]

Problem

I realise the specification of an MPI ABI is a very complicated endeavour, but as a very small step in this direction, I would like to propose that there be an ABI for identifying a particular MPI implementation.

This is important when using MPI with other languages: I am one of the maintainers of the Julia language MPI bindings, and one of the biggest difficulties is identifying which implementation of MPI is being used. At the moment, we compile a small C program to identify the MPI implementation and the necessary constants, but this has many drawbacks in terms of usability (namely that it requires a C compiler, and makes it difficult to switch between versions).

MPI_Get_library_version can currently be used to identify an implementation, but it is not well specified as a C ABI as it is not possible to determine the following without parsing a header file:

1. the size of the buffer to allocate (i.e. the value of MPI_MAX_LIBRARY_VERSION_STRING from the header file)
2. if a non-default calling convention is used (e.g. Microsoft MPI uses stdcall)

Proposal

Point 1 could easily be addressed by adding in the specification the maximum value of MPI_MAX_LIBRARY_VERSION_STRING: the largest value I have seen is 8192 used by mpich.

Point 2 is more difficult to standardize, since it is much more platform specific. However it only affects one implementation on one platform, and on that platform it appears to be the dominant implementation, so this is less of an issue. If anyone does have a suggestion for solving this, I would be grateful.

Changes to the Text

MPI_MAX_LIBRARY_VERSION_STRING must be a value less than or equal to 8192.

Impact on Implementation

As far as I know, this should not affect any implementations. If it does, the value can be chosen to be larger.

Impact on Users

This will make it easier to develop MPI bindings in languages other than C or Fortran, which in turn should improve user experience.

References

There is some discussion of this problem in JuliaParallel/MPI.jl#169.

[jsquyres]

Greetings Simon; thanks for the suggestion. I wonder if you can clarify something for me, because I'm not entirely sure I understand the problem you're trying to solve.

You want to fix the length of the string that is returned by MPI_Get_library_version() to X (doesn't matter what the value of X is). Can you explain why a compile-time constant isn't sufficient for that?

Specifically: I'm trying to understand why it would be necessary to have this as a fixed length. Are you trying to ship a binary with a fixed-length buffer for calling MPI_Get_library_version()? If not -- i.e., if you're compiling from source -- is there a problem with using MPI_MAX_LIBRARY_VERSION_STRING to size the buffer properly?

[jeffhammond]

Are you trying to ship a binary with a fixed-length buffer for calling MPI_Get_library_version()?

@jsquyres They want to get away from requiring a C compiler...

At the moment, we compile a small C program to identify the MPI implementation and the necessary constants, but this has many drawbacks in terms of usability (namely that it requires a C compiler, and makes it difficult to switch between versions).

[simonbyrne]

You want to fix the length of the string that is returned by MPI_Get_library_version() to X (doesn't matter what the value of X is).

To clarify: I don't want to fix the length, I just want to specify an upperbound, so that allocating a buffer of that length would be sufficient and guaranteed not to cause a buffer overflow. Since the C version of MPI_Get_library_version is required to return a null-terminated string, passing a buffer that is too big is not an issue. This should not require any changes to existing MPI implementations.

Can you explain why a compile-time constant isn't sufficient for that?

Specifically: I'm trying to understand why it would be necessary to have this as a fixed length. Are you trying to ship a binary with a fixed-length buffer for calling MPI_Get_library_version()? If not -- i.e., if you're compiling from source -- is there a problem with using MPI_MAX_LIBRARY_VERSION_STRING to size the buffer properly?

The constant MPI_MAX_LIBRARY_VERSION_STRING is specified in the header file, which requires a C compiler to parse. Julia, like many other recent languages, provides a foreign function interface (FFI) for calling C-compatible functions directly, without invoking a C compiler.

Unfortunately in the case of MPI this is complicated by the lack of a standard ABI (e.g. the size and alignment of types like MPI_Comm can vary), so at the moment we generate a small C program that in turn generates the necessary Julia code to define these types and values of constants. However this causes frequent problems when users want to switch MPI implementations, or have nonstandard MPI or compiler configurations.

Now that most MPI installations are ABI compatible with one of MPICH, OpenMPI or Microsoft MPI, if we can identify the implementation without needing to invoke a C compiler, then we can provide pre-generated files specifying the necessary types and constants, and fallback to our current code generation machinery only in the cases where it does not conform to one of those.

[omor1]

An alternative could be a query function MPI_GET_MAX_LIBRARY_VERSION_STRING that returns the MPI_MAX_LIBRARY_VERSION_STRING. This inflates the number of functions that MPI implementations must support, but is a trivial function to implement.

int MPI_Get_max_library_version_string(int *length)

MPI_Get_max_library_version_string(length, ierror)
    INTEGER, INTENT(OUT) :: length
    INTEGER, OPTIONAL, INTENT(OUT) :: ierror

MPI_GET_MAX_LIBRARY_VERSION_STRING(LENGTH, IERROR)
    INTEGER LENGTH, IERROR

[simonbyrne]

I think if larger changes are under consideration, I would prefer something that could address point 2 as well: one example would be making the version string available via an extern char array.

[jsquyres]

Ok, that's a fair point: you want to be able to have some information that is available to C and Fortran at compile time, but is not necessarily available at run time (e.g., in Open MPI's case, MPI_MAX_LIBRARY_VERSION_STRING is a #define in C).

To be more multi-language friendly, I'd actually take @omor1's proposal and raise you: have a symbol that can be used to find all such compile-time values (not just MPI_MAX_LIBRARY_VERSION_STRING). Either some type of query function (that doesn't rely on enums or other compile-time constants itself so that it can unconditionally be invoked from other languages without a priori knowledge of the MPI implementation), or a struct full of members, or perhaps changing the requirements of the various compile-time constants to also be run-time values. Regardless of the mechanism, I'd propose fixing this for all the values that you need -- not just MPI_MAX_LIBRARY_VERSION_STRING.

1. Query function: perhaps something like MPI_Query_constant(const char *name, int *value) (that's off the top of my head -- someone should think that through to make sure it covers all cases properly).
2. Struct full of members. This is probably a bit unwieldy, but perhaps something like

struct MPI_Constants_t {
    MAX_LIBRARY_VERSION_STRING,
    // ...others
} MPI_Constants = {
    .MAX_LIBRARY_VERSION_STRING = implementation_specific_value,
    // ...others
}

1. Convert compile-time constants to also be run-time values. This is probably the least-desirable option.

---

However this causes frequent problems when users want to switch MPI implementations, or have nonstandard MPI or compiler configurations.

I hear you on this one: the situation just plain sucks. There's unfortunately too much historical baggage here -- MPI was not designed with an ABI in the beginning; it's hard to reverse-graft an ABI on to 25+ years of software and design. MPI v4.0 is all but in the bag, and ABI didn't make the cut. I think the next best hope is MPI-5.0, and the potential for more-than-C-and-Fortran native bindings.

[simonbyrne]

Thanks @jsquyres, that would work for the constants C ints, but not say arbitrary handles (though you could provide the Fortran constants and call F2C conversions, but would need to know the types), and pointers like MPI_BOTTOM.

[jsquyres]

I don't quite understand, probably because I don't know enough about how other languages (e.g., Julia) work.

The MPI handles (e.g., MPI_COMM_WORLD) are defined to be run-time values. Open MPI's handles are pointers to static objects, and are therefore effectively only retrievable at run-time. MPICH's handles just happen to be integers. Regardless, this scheme was not designed such that applications should know or care what the value of these handles are.

My assumption is that you are looking for a way to get the values of these handles into Julia -- correct? If so, is there an interact-with-C mechanism in Julia such that you can effectively either bind a Julia value to a C value (like Fortran), or at least load a Julia value from a C value?

[simonbyrne]

The MPI handles (e.g., MPI_COMM_WORLD) are defined to be run-time values. Open MPI's handles are pointers to static objects, and are therefore effectively only retrievable at run-time. MPICH's handles just happen to be integers. Regardless, this scheme was not designed such that applications should know or care what the value of these handles are.

My assumption is that you are looking for a way to get the values of these handles into Julia -- correct? If so, is there an interact-with-C mechanism in Julia such that you can effectively either bind a Julia value to a C value (like Fortran), or at least load a Julia value from a C value?

We have figured out a reasonably robust way of getting the values of the handles: we determine the Fortran constants (which are C integers), then at MPI.Init we call MPI_XXXX_f2c to convert it to a C handle (the Rust MPI bindings appear to do something similar). The bigger problem is figuring out the appropriate size and alignment of the handles themselves (which at the moment requires parsing the C headers).

[omor1]

which at the moment requires parsing the C headers

There was a discussion previous in #137 regarding whether C MPI programs that do not include the mpi.h header are valid, with the general consensus that this was not the case.

[jsquyres]

I think that the Forum is amenable to keeping the door open to other languages. Indeed, there's groups working on opening the door wider in future revisions of the MPI spec. So let's keep having the conversation here... 😄

In Julia, is there no way to get the size of a C type? (again, forgive my ignorance here)

What do you need the alignment for? Julia shouldn't be creating new MPI handles -- you should always be getting them from C, right?

[simonbyrne]

In Julia, is there no way to get the size of a C type? (again, forgive my ignorance here)

Julia knows the sizes of the builtin C types, and you can construct Julia types which match arbitrary C types in size and alignment, but there is no way to figure out how a C type is defined without parsing the header (which requires a C compiler).

What do you need the alignment for? Julia shouldn't be creating new MPI handles -- you should always be getting them from C, right?

Alignment is dependent on how the handle type is defined (e.g. a struct containing two 32-bit integers has different alignment than a 64-bit pointer).

My point was that if you want to make the ABI queryable, you would need to know both size and alignment of the various handles. You could expose these via the proposed MPI_Query_constant by defining additional names to query these, e.g. MPI_COMM_size and MPI_COMM_alignment. I'm not sure how one would handle cases like MPI_BOTTOM: I suspect an additional function would be required.

That said, I think a more robust solution would be to define a mechanism by which implementations could identify their ABI, and then leave it up to them to specify it how they wish. MPI_Get_library_version seemed like the simplest mechanism to achieve this, but I could see the utility in another (e.g. ideally the various MPICH-compatible implementations would all identify themselves the same way).

[simonbyrne]

On a related note: another recent pain point we were experiencing is that there does not appear to be a consistent mechanism for determining whether an MPI implementation supports the CUDA-aware interface (OpenMPI provides MPIX_Query_cuda_support, but I haven't seen anything similar in other implementations). Providing a mechanism to query such features would seem related to this issue.

[omor1]

I think that the Forum is amenable to keeping the door open to other languages. Indeed, there's groups working on opening the door wider in future revisions of the MPI spec. So let's keep having the conversation here... 😄

Agreed that this is an important conversation—just pointing out that the current standard makes that somewhat difficult.

I know that when I was working on a binding for Apple's Swift language around three years ago I basically resorted to wrapping around all the functions and constants with inline functions so as to force a common ABI (the Swift–C interface had issues with function-like macros in Open MPI and imported enums as the Swift type Integer rather than CInt, the former being a word-length integer and the latter the same as the C int type).

I don't know if a similar technique works for Julia, since it seems that its C integration uses a different mechanism than Swift.

there does not appear to be a consistent mechanism for determining whether an MPI implementation supports the CUDA-aware interface

Any GPU interoperability is incredibly implementation-specific and not specified by the standard. The are currently no proposals to add such support to the standard as far as I know. Querying for extensions might be a useful addition actually, but that should definitely be discussed in a different issue.

A workaround might be to use dlsym to determine whether the MPIX_Query_cuda_support symbol exists and therefore can be called.

[omor1]

Regarding C MPI handles—the standard actually restricts how they can be implemented. On page 13 (MPI-3.1 §2.5.1) the following is stated:

The C types must support the use of the assignment and equality operators.

The only types in C that supports these are pointers and arithmetic types. Now I suppose an implementation could technically use long double _Complex and still be standards-compliant, but I'm fairly certain that all implementations use either pointers or integers.