osc rdma hang with multiple windows

[markalle]

Using the testcase mt_1sided.c (and its support files 1sided.c mt_1sided_td1.c mt_1sided_td2.c) from this test harness pull request:
https://github.com/open-mpi/ompi-tests/pull/25
I get a hang from the osc rdma component:

% mpicc -o x mt_1sided.c mt_1sided_td1.c mt_1sided_td2.c
% mpirun -host hostA,hostB -mca osc rdma -mca pml ob1 -mca btl openib,self,vader ./x

This was from a vanilla build of openmpi-master-201612022109-0366f3a.

The single threaded "1sided.c" passes. And I think the key difference in the multi-threaded version is that there are two windows active (one for each thread).

1sided.c is a fairly broad test, looping over several synchronization types as well as contiguous and non-contiguous datatypes. I could probably whittle the test to be a bit more targeted for this particular hang if needed.

[jsquyres]

@hjelmn Can you have a look?

[rkowalewski]

@markalle Your link url does not work. We have probably the same issue regarding communication progress with multiple windows in our project. Could you please fix your url since I want to verify my assumption.

[jsquyres]

@rkowalewski Our ompi-test repository is not public; that's why the link is broken for you.

@markalle Are you amenable to sharing your new tests with @rkowalewski? You could post them in a gist, or attach them here on this issue.

[rkowalewski]

@jsquyres @markalle I spent more time with this. Finally, I have pinpointed the bug and created another issue: #2538
Maybe there is some relation between these two issues.

[markalle]

@jsquyres @rkowalewski Yeah, since we put it in the ompi-tests it's considered public-ish at least. Even the stuff I've checked in there is a whittled down version of our original tests, as giving out these smaller somewhat whittled down tests is easier to get management approval for.

[markalle]

Here, see if I made this gist right:
https://gist.github.com/markalle/56f739c3c907e53960d22d5de2fe4bc9
It should have 1sided.c, mt_1sided.c, mt_1sided_td1.c and mt_1sided_td2.c.

[jsquyres]

Perfect -- thanks @markalle!

[hjelmn]

Please test with the latest 2.0.2 release candidate.