request attributes

[jeffhammond]

Problem

We have attributes on communicators, windows and datatypes but not other handles. There are use cases for request attributes.

For example, if I want to allocate temporaries associated with a nonblocking operation, there is no way to deallocate those temporaries when the operation is completed. Request attributes would allow me to cache the buffer address on the request and clean it up when I call wait.

Proposal

Add requests to the Caching subsection.

Changes to the Text

The copy-and-paste for this is pretty obvious.

Impact on Implementations

I've looked at MPICH as an example of the implementation impact. They would need to add a single pointer field to the internal request object. This struct already contains device-specific and other conditional members so its size is already not constant.

struct MPIR_Request {
  ...
  struct MPIR_Attribute * attributes;
  ...
}

None of the performance critical code paths will touch this field, and depending on where it is in the internal request struct, it may not even load into cache until explicitly accessed.

Impact on Users

This enables a number of interesting library features.

For example, in VAPAA, I can allocate a temporary buffer to support otherwise intractable features like noncontiguous subarrays with user-defined datatypes. Similarly, other Fortran interfaces can use this to avoid having to add intrusive changes related to CFI_cdesc_t deep inside of the implementation for unlikely but required usages.

Another use case is in tools. Tools can attach an attribute that is a function pointer to a request and then fire that callback in the tool interposition code inside of request synchronization.

References and Pull Requests

#663

[jeffhammond]

Implementation prototype here: https://github.com/jeffhammond/mpich/tree/request-attributes-prototype

[softwaretraff]

see the EuroMPI paper

Jesper Larsson Träff, Ioannis Vardas:
Library Development with MPI: Attributes, Request Objects, Group Communicator Creation, Local Reductions, and Datatypes. EuroMPI 2023: 5:1-5:10

for discussion and use-case why this might be a good idea (also something on 3-argument local reduction and typed memory copy)

Jesper