Also run the CI tests with shared windows disabled

[fuerlinger]

Once DART successfully builds again with disabled shared windows, we should run CI tests with both configurations: enabled and disabled.

Disabled shared windows more closely resembles the distributed memory setting and might uncover issues that would otherwise go unnoticed. Plus it is needed for NastyMPI to work (intercept puts and gets).

[fmoessbauer]

I agree, that this should definitely be tested. However this (again) rises the time the CI takes to finish.

Implemented in branch feat-184-ci-sw.

[fmoessbauer]

On CircleCI the tests fail using the OpenMPI2 container with more than one unit: See CI Output.

@fuchsto please have a look at the logs.

[fuchsto]

It's fixed in branch feat-52-nastympi, will be fixed with merge of PR #170

See branch feat-184-shwin

[fuchsto]

@rkowalewski @fmoessbauer @fuerlinger

In PR #189, I re-enabled MPI shared windows in CI builds as there apparently is a bug in OpenMPI 2, if I'm not mistaken.

There is a deadlock where one unit is entering a barrier, and another unit calls MPI_Put to a window shared with the barrier'ed unit (but in it's own segment):

This should not block, if I understand the MPI standard correctly.

For the full log, see CircleCI build no. 646:
https://circleci.com/gh/dash-project/dash/646

Note that this failed for OpenMPI 2 only.
For MPICH and OpenMPI 1, the Minimal and NastyMPI test configurations passed.

This defect has could not be provoked by NastyMPI, another hint that it's actually a bug in OpenMPI 2.

[rkowalewski]

@fuchsto Actually you have been a bit faster than me. You are totally right, OpenMPI-2 blocks in one-sided communications. However this issue happens only if we use multiple MPI windows which is actually the case in DART. The issue has been recently discovered and is documented in Issue #2530 in the Open-MPI Repository.

Additionally, we have another bug in dart_exit. It is a memory leak since we do not free the allocated memory for local memory allocations. Surprisingly,
this causes a segmentation fault in Open-MPI 2 when MPI_Finalize is executed. Actually,
all test cases have never been successfully terminated. I fixed this in a recent commit. Additionally I created an issue for Open-MPI since this segmentation fault should, of course, not happen.

[fuchsto]

Okay, so long story short:

We won't use OpenMPI 2 in CI for now.

This is just one of several arcane defects in OpenMPI 2 I stumbled upon and fixing them doesn't help DASH/DART at all.

[rkowalewski]

We should revert to OpenMPI 1.10.x

[fmoessbauer]

Yes and No - I recommend the following:

• We keep the OpenMPI2 container (but do not run in CI) and from time to time we can manually test if OpenMPI2 works for us.
• We update the OpenMPI container to OpenMPI 1.10.latest

This is also beneficial, because if we only use only two Circle containers, we can run two commits at a time 👍

[fuchsto]

Yes, that's my plan. Once they got a grip on RDMA again, we will use it.
Good news is that we only need two CircleCI containers now.

... emm, as you already mentioned.

[fuchsto]

Fixed in #189