Alternative destination-specific serialization

[mrocklin]

In some cases objects may want to serialize themselves dependent on the sender and recipient machine. One such example is multi-gpu settings where certain workers may be able to exchange GPU data efficiently if they are on the same physical host or nearby in the network.

To meet this goal it may make sense to enable host-specific serialization, that is we would optionally use a new serialization function that was invoked for certain types:

serialize(obj, host, destination) -> bytes

We would probably have to register serialization functions for certain types or else look for a special protocol like __getstate_host__ and __setstate_host__.

cc @seibert @sklam @gforsyth

[sklam]

IIUC, a usecase for GPU array would be: serialize(gpuarray, host, dest) send a handle for the destination to request for direct GPU transfer without going thought pickling and the network.

Is that right?

If so, who is reponsible to keep obj alive so that it is available when the destination request for the peer transfer?

[seibert]

that is a good point. Any of these alternate-channel communication methods require some resource on the source to exist until the destination has completed the transfer. This will require the destination to ACK back to the source before that resource can be released.

[mrocklin]

That will commonly be true in practice but is hard to guarantee. One could hack this by implementing a bit of reference counting in the serialize function.

[mrocklin]

As brought up in #614 (comment) this would also be helpful for clusters with heterogeneous language workers (Python worker sending data to Julia worker)

[pitrou]

So, this is a feature baked in the new I/O infrastructure, since each transport (or backend) decides on the serialization it wants to use. For example, the inproc backend defined in #887 uses no serialization: Python object references are simply queued from one thread to another.

[mrocklin]

So here are a couple options:

1. We include source and destination host to serialization functions. They send enough data to the other side to initiate an efficient communication back through some side channel.
2. We create different comm types for inter-host and intra-host communications (see Intra-host communication #2046) and then allow serialization to differ among comm types. Serialization schemes do not get source and destination host information, but now we can different kinds of serialization schemes for intra-node and inter-node, and use them appropriately for inter-node comms and intra-node comms.

For GPU transfers in either case the data serialized is just some connection and address information. When the receiving side accepts this data and deserializes it it initiates a side-channel communication. We allow serialization functions to request that the sending side only release resources after the receiving side has finished the deserialization process (which in this case is a side-channel transfer)

[jakirkham]

Is this still relevant?