a problem in the btl/sm shared_mem_btl_rndv.HOSTNAME

[sekifjikkatsu]

I think there is a problem in the btl/sm component.
It is the synchronization problem that is between the node master process and other child processes.
Other child processes read 0 byte from shared_mem_btl_rndv.HOSTNAME file before the node master
process writes sizeof(opal_shmem_ds_t) bytes to the file.
I think this problem can be solved with the following correction.

Index: ompi/mca/btl/sm/btl_sm.c
===================================================================
--- ompi/mca/btl/sm/btl_sm.c    (リビジョン 4013)
+++ ompi/mca/btl/sm/btl_sm.c    (作業コピー)
@@ -180,6 +180,7 @@
 static int
 sm_segment_attach(mca_btl_sm_component_t *comp_ptr)
 {
+    struct stat buf;
     int rc = OMPI_SUCCESS;
     int fd = -1;
     ssize_t bread = 0;
@@ -195,6 +196,14 @@
         rc = OMPI_ERR_IN_ERRNO;
         goto out;
     }
+    do {
+        if (0 != fstat(fd,&buf)) {
+            opal_output(0, "sm_segment_attach: "
+                           "fstat errno=%d\n",errno);
+            rc = OMPI_ERROR;
+            goto out;
+        }
+    } while (sizeof(opal_shmem_ds_t) != buf.st_size);
     if ((ssize_t)sizeof(opal_shmem_ds_t) != (bread =
         read(fd, tmp_shmem_ds, sizeof(opal_shmem_ds_t)))) {
         opal_output(0, "sm_segment_attach: "

This problem occurs in the Open MPI version 1.8.
And it may not occur in the Open MPI master.

[ggouaillardet]

should the test be ?
while (sizeof(opal_shmem_ds_t) > buf.st_size)

from create_rndv_file if type is MCA_BTL_SM_RNDV_MOD_MPOOL, some extra stuff is written into the file.

that being said, i am a bit puzzled by this.

this solves the case in which local task 0 created the file, but did not write yet.

assuming we run into this case, it means task > 0 could try to open the file before it was created by task 0, and hence fail ... i would suspect higher level ensure we do not run into this in the first place.

btw, which filesystem is used by the sm btl ?
local filesystem ? tmpfs ? /dev/shm ? network filesystem ?
if this is a network filesystem, did you get a warning about that ? (or did you configure ompi not to issue a warning ?) the recommendation is not to use a network filesystem for this.

[sekifjikkatsu]

Thank for the comment.

The file which is opened in sm_segment_attach is MCA_BTL_SM_RNDV_MOD_SM.
So, I think that size of the file is sizeof (opal_shmem_ds_t).

And, I agree to test as following,

while (sizeof(opal_shmem_ds_t) > buf.st_size)

I think that task > 0 does not open the file before local task 0 creates the file.
Task > 0 wait for the file becomes R_OK state by access() in setup_mpool_base_resources.

     /* Wait for the file to be created */
    while (0 != access(comp_ptr->sm_rndv_file_name, R_OK)) {
        opal_progress();
    }

The sm btl uses /dev/shm. it is not a network filesystem.

[ggouaillardet]

@sekifjikkatsu Thanks for pointing 2), now this makes sense to me, and there is clearly a race condition.

i can see three options

1. while ((sizeof(opal_shmem_ds_t) + out_res->size) > buf.st_size)
   /* i previously missed the second read, hence out_res->size , also, opal_progress() should be invoked if the size is too small */

2. replace the access test by stat and also check the size

3. create the file with a name specific to local task 0, write its content, and then hard link it to the name that is used by local task > 0

any preference ?

@hjelmn @bosilca any thoughts ? (according to owner.txt, UTK is the maintainer, but i am not sure of who is maintaining this module)

[jsquyres]

(...I'm only loosely reading this issue...)

It sounds like there's an assumption in the >0 processes that the =0 process will atomicly open and write all N bytes at once -- which may not be true (and therefore there's a race condition).

Would it be possible to have the =0 process open a different file, write the contents, and then rename(2) the file to be the file that the >0 processes are expecting? This would guarantee that, from the >0 process' perspective, the file atomicly appeared and had the correct contents.

[ggouaillardet]

@jsquyres using rename(2) is a simpler and better variant of my third option.
I will make the prs from Monday unless there are any objections.

[bosilca]

There are 2 logical steps in here, separated by a fence (the modex operation). On each node the local master creates and initializes the shared memory files while in function create_rndv_file. This happen before the call to the modex, so there are no possible race conditions as the other processes are already blocked in the modex exchange (PMIX fence in ompi_mpi_init.c:644). Once the local master joins the modex, everybody completes and all remaining local processes continue with attaching to the shared file.

[rhc54]

What happens when we set mpi_add_procs_cutoff=0 (or exceed the cutoff point)? Under that condition, we don't execute a fence in mpi_init - we just fall thru. Do we need something to avoid the race condition in that case?

[bosilca]

If we don't have a fence in MPI_Init, then we do need to protect the creation of the shared file by other means. However, even when I set ompi_mpi_add_procs_cutoff to 0 I still see the fence at ompi_mpi_init.c:644.

[rhc54]

My bad - the cutoff is for dynamic add procs. The correct mca param is pmix_base_async_modex=1