16-bit floating-point support for C/C++

[jeffhammond]

Problem

There is interest in supporting 16-bit floating point (henceforth FP16) in MPI.

See https://lists.mpi-forum.org/pipermail/mpiwg-p2p/2017-June/thread.html

Proposal

Add a type associated with FP16 that does not depend on the Fortran definition (MPI_REAL2).

See references. Various non-standard names for FP16 including __fp16 and short float. The candidate ISO name is _Float16. It may be prudent for MPI to add a type (along the lines of MPI_Count and MPI_Aint) since ISO C and C++ have not standardized names yet and they may not be identical; the typedef would be MPI_Float16, which it may be deprecated as soon as there is an ISO C/C++ name.

Changes to the Text

TODO

Impact on Implementations

The implementation of FP16 is straightforward, following whatever code exists for MPI_REAL2 today, or by copying code for FP32 with s/32/16/g.

A high-quality implementation may need to use special care when implementing reduction operators that can lose precision.

Impact on Users

FP16 support will be available independent of anything related to Fortran.

Users working on machine learning do not use Fortran anywhere (except perhaps indirectly in BLAS) and are not likely to be satisfied with MPI_REAL2, particularly since an implementation can omit support for it if a Fortran compiler is not present.

References

• Half-precision floating-point format on Wikipedia.
• ISO/IEC JTC 1/SC 22/WG14 N1945 (ISO C proposal)
• ISO/IEC JTC1 SC22 WG14 N2017 (ISO C++ proposal)
• GCC documentation for Half-Precision Floating Point and Additional Floating Types (e.g. _Float16)
• Clang/LLVM _Float16 support for C/C++ commit
• Intel® Half-Precision Floating-Point Format Conversion Instructions
• Performance Benefits of Half Precision Floats

[ahori]

Attached
FP16-Reading-1st.pdf
is my very first draft for the standard with FP16. Modified parts are colored with cyan. My points are;

1. MPI_HALF, MPI_C_HALF_COMPLEX, MPI_CXX_HALF_COMPLEX, MPI_HALF_INT (reduction), MPI2_HALF (reduction) are added (MPI_REAL2 and MPI_COMPLEX4 were already defined).
2. These new FP16 related data types are optional.

Any comments and/or suggestions are welcome.

[dholmes-epcc-ed-ac-uk]

Is the MPI_CXX_* type aimed at providing C++ bindings? That support has been removed from MPI and should not now be updated or extended.

MPI2_HALF looks like a typo for MPI_HALF2?

If we have MPI_C_HALF_COMPLEX (for C) do we need MPI_F_HALF_COMPLEX (for Fortran) and MPI_*_HALF_COMPLEX2 (for reductions, in both languages)?

How should a user program determine if an MPI library supports these types if they are optional? Should there be a mandatory compile-time constant like MPI_HAS_HALF_TYPES?

#ifdef MPI_HAS_HALF_TYPES
MPI_HALF myVar;
#else
MPI_FLOAT myVar;
#endif

[kawashima-fj]

@dholmes-epcc-ed-ac-uk My understanding is that the C++ binding is removed but using the C binding from C++ programs is still supported. C++ datatypes are required for such case. Actually MPI_CXX_BOOL was added in MPI-3.0 which removed the C++ binding.

[bosilca]

@dholmes-epcc-ed-ac-uk I assume MPI2_HALF was indeed a typo for the reduction type MPI_2HALF. So far optional types were just not in the headers, the standard assumed it was the developer responsibility to detect the lack of such types by whatever means they want/need.

@ahori what is the expected link between the newly added half precision type and MPI_REAL2 and MPI_COMPLEX4 ? Are modern compilers translating Fortran REAL*2 type into half precision ?

[jeffhammond]

@dholmes-epcc-ed-ac-uk

Is the MPI_CXX_* type aimed at providing C++ bindings? That support has been removed from MPI and should not now be updated or extended.

We deprecated the C++ functions. We still need C++ types, for example, because C99 _Complex T and C++ std::complex<T> are not the same type.

If we have MPI_C_HALF_COMPLEX (for C) do we need MPI_F_HALF_COMPLEX (for Fortran) and MPI_*_HALF_COMPLEX2 (for reductions, in both languages)?

MPI_COMPLEX2 is the Fortran complex type consisting of two 16-bit floating-point numbers.

How should a user program determine if an MPI library supports these types if they are optional?

The same way you determine whether the 14 optional types we already have in Table 13.2 are supported.

[jeffhammond]

@ahori We should use the ISO C candidate name _Float16 instead of half or floathalf, unless we want to define a placeholder to deal with the potential differences between C and C++. For that, I'd choose something like MPI_Float and provide the text explaining that it is a typedef to the IEEE half precision type supported by the compiler.

[ahori]

@dholmes-epcc-ed-ac-uk

MPI2_HALF looks like a typo for MPI_HALF2?

Correct.

If we have MPI_C_HALF_COMPLEX (for C) do we need MPI_F_HALF_COMPLEX (for Fortran) and MPI_*_HALF_COMPLEX2 (for reductions, in both languages)?

I just added FP16 types almost automatically without any intention. I have noticed that there is no explanation on MPI_C_* and MPI_CXX_* types found in the MPI-IO external 32 section. I checked MPI 2.2 and I found these were added since 2.2. Is there anybody who can explain what they are?

How should a user program determine if an MPI library supports these types if they are optional? Should there be a mandatory compile-time constant like MPI_HAS_HALF_TYPES?

No idea. But the same thing happens on the other optional types. (configure can detect :-)

@jeffhammond

We should use the ISO C candidate name _Float16 instead of half or floathalf, unless we want to define a placeholder to deal with the potential differences between C and C++. For that, I'd choose something like MPI_Float and provide the text explaining that it is a typedef to the IEEE half precision type supported by the compiler.

I have just remembered what Rolf suggested to use "short float" instead of "half." He also suggested to have MPI_FLOAT*_T types as below;

Named Predefined Datatypes | C types
---------------------------+------------
MPI_SHORT_FLOAT | short float
MPI_FLOAT16_T | float16_t
MPI_FLOAT32_T | float32_t
MPI_FLOAT64_T | float64_t

The traditional type names such as MPI_DOUBLE cannot be deprecated just for backward compatibility. Anyway I will update the text in a few days, before flying to Denver.

@bosilca

what is the expected link between the newly added half precision type and MPI_REAL2 and MPI_COMPLEX4 ? Are modern compilers translating Fortran REAL*2 type into half precision ?

I think/suppose/hope so.

[ahori]

@ahori

If we have MPI_C_HALF_COMPLEX (for C) do we need MPI_F_HALF_COMPLEX (for Fortran) and MPI_HALF_COMPLEX2 (for reductions, in both languages)?
I just added FP16 types almost automatically without any intention. I have noticed that there is no explanation on MPI_C and MPI_CXX_* types found in the MPI-IO external 32 section. I checked MPI 2.2 and I found these were added since 2.2. Is there anybody who can explain what they are?

Oh, I got the answer of this. There 2 ways of expressing complex numbers in C++, _Complex. and std::complex. MPI_CXX_*_COMPLEX means std::complex, whereas MPI_C_*_COMPLEX means _Complex. To distinguish these, we need both.

[ahori]

@kawashima-fj

My understanding is that the C++ binding is removed but using the C binding from C++ programs is still supported. C++ datatypes are required for such case. Actually MPI_CXX_BOOL was added in MPI-3.0 which removed the C++ binding.

Oh, why do we need C++ types ? What happens when a C++ object appears in the argument of MPI_Send or MPI_Recv ? I believe this is not allowed. Is there any situations where a C++ object can be the argument of an MPI function ?

[kawashima-fj]

@ahori

Accessing the C binding from C++ is allowed.

from MPI-3.1 p.5:

1.8 Who Should Use This Standard?
This standard is intended for use by all those who want to write portable message-passing
programs in Fortran and C (and access the C bindings from C++).

from MPI-3.1 p.36:

MPI requires support for inter-language communication, i.e., if messages are sent by a
C or C++ process and received by a Fortran process, or vice-versa. The behavior is defined
in Section 17.2.

And as you mentioned, C++ has std::complex<float> etc., which is not available in C and is not identical to C float _Complex. Therefore we need C++ specific datatypes to send/receive objects of such type. If short float _Complex is defined in C and std::complex<short float> is defined in C++ and they are not identical, we need MPI_C_SHORT_FLOAT_COMPLEX and MPI_CXX_SHORT_FLOAT_COMPLEX (or other names).

I don't mean sending/receiving C++ class object.

[ahori]

@kawashima-fj

According to here;

http://en.cppreference.com/w/cpp/numeric/complex (in the box titled "Non-static data members")

the layouts on complex numbers of C++ and C are (guaranteed to be?) the same.
So, here is my point;

• If std::complex is implemented as a REAL object, then MPI_CXX_*_COMPLEX types are needless.
• If std::complex is NOT a real object and its data layout is the same with C, then again MPI_CXX_*_COMPLEX types are needless.
• (If std::complex is NOT a real object and its data layout is NOT the same with C, then who wins ?)

Further, I think the same thing happens on bool types.

Am I missing something?