MPI_Waitall optimization

[ompiteam]

The current MPI_Waitall() has O2 complexity
can do better.
very important for pps benchmarks

When using MPI_ISend + MPI_Waitall() is worse than using MPI_Send

[ompiteam]

Imported from trac issue 3663. Created by miked on 2013-07-11T09:20:07, last modified: 2014-04-22T16:22:58

[ompiteam]

Trac comment by bosilca on 2013-07-11 12:31:39:

Is there any proof the performance impact is coming from the MPI_Waitall implementation instead of the lack of ordering at the network level or the ordering (and amount) of the memory registrations) ?

[ompiteam]

Trac comment by amikheev on 2013-07-12 01:41:13:

Benchmark (slighly modified osu_mbw_mr) was done with zero sized messages. Either eager rdma or eager transport was used. It rules out memory registration.

Ordering can be ruled out because we essentially used single qp and single transport.

Switching from MPI_Waitall to MPI_Wait() on every single request improved things.

[ompiteam]

Trac comment by jsquyres on 2013-07-12 06:20:25:

Are you guys working on a patch?

[ompiteam]

Trac comment by jsquyres on 2013-07-16 12:12:40:

I notice this is assigned to ompi-rm-1.7, but there's no patch or r number associated with this ticket.

Assigning back to miked.

[ggouaillardet]

@miked-mellanox is the issue still here ?

if yes, could you post the program you use to evidence it and how you run it ?

[mike-dubman]

Hi @ggouaillardet
some of the issues we fixed in OMPI, but not all.
See attached ppt presented in one of the OMPI dev meeting with details.

We may have internal patch for MPI_Waitall, @alex-ma - please comment.

The test used was osu_mbr_mr

@alex-mikheev - do you have something to add?

[alex-mikheev]

@ggouaillardet @miked-mellanox we never had a patch for MPI_waitall(). Instead we changed benchmark code to use MPI_Wait() to pick request completion in order instead of waiting for the whole window. I recall it improved message rate by 5-10%

The benchmark in question is osu_mbw_mr. I will see if I can upload my changes to github

[jsquyres]

This issue (improving WAIT/TEST) has come up a few times (e.g., in the face-to-face dev meeting in Jan and June 2015). @bosilca is working on it.

[jsquyres]

@bosilca Did you ever make any progress on this TEST/WAIT optimization stuff?

[bosilca]

Yes. It is in paper submission stage ...

[mike-dubman]

out of curiosity, could you please share offline the conclusions and results?

[bosilca]

This issue is now stale, all future discussion will go in #1820 ,