ompi_mpi_instance_init_common needs to be local

[hppritcha]

ompi_mpi_instance_init_common is invoked when an application uses either MPI_Init or MPI_Session_init. The later function has purely local semantics to the invoking application. As is though, this function has to have a PMIx fence involving all procs in a "job" as some PMLs require this for the add_procs method to work correctly, even if just the "local" procs are included in the argument array.

This call and the preceding PMIx Fence need to be moved into the MPI_Init specific code. Code needs to be added in the MPI_Comm_create_from_group and MPI_Intercomm_create_from_groups to do the add_procs at that point. This may require either new PML methods or modification to add_proc methods that don't check for previously added procs.

[hppritcha]

related to #11212

[rhc54]

@hppritcha Just wondering: I have been working on a side-branch to ensure that PMIx_Group_construct properly returns all the job-level info and modex info for all group participants. If that is correctly completed, then would that remove this problem? Just trying to understand why "add_procs" would require a fence operation if all the job-level and modex info were known.

[hppritcha]

i'm not sure that would help. i'd need to double check but i'm pretty sure one thing that messes things up is the way smsc tries to get itself set up using the IMMEDIATE attribute for a pmix_get operation.

[rhc54]

Precisely - immediate means "check the server and return not found if the server doesn't have it". Right now, the server will not have the info because the group construct isn't correctly returning it when completing the operation. Hence my question. If "add_procs" requires a fence for some reason other than needing endpt info, then it won't help. But if not finding info is the only reason it is failing, then fixing the group construct is the right answer.

[rhc54]

one thing that messes things up is the way smsc tries to get itself set up using the IMMEDIATE attribute for a pmix_get operation.

Had a few minutes and took a peek at that component, and you are correct. The way that was done forces a "fence" to be called prior to getting the endpt.

What you want to do is remove the "immediate" attribute. The PMIx library will call up to the server to see if the value is available there. Since the peer is a local one, the PMIx server will not pass the request up to the host daemon for a "dmodex" operation. Instead, it will wait for the local peer to commit it (the "immediate" attribute causes the server to return "not found" without waiting).

You technically don't need a timeout, but if you want to ensure you don't hang in the component (e.g., if the peer never provides its info for some reason), then just replace the "immediate" attribute with a "timeout" and provide some reasonable value (e.g., 2 secs). You'll probably have to just directly call PMIx_Get, or else create a new OPAL macro (overkill unless someone else needs it) to add the timeout.

[hppritcha]

thanks for the info. I wonder why that IMMEDIATE option was being used in the first place.

[rhc54]

In fact, I think the way that component (and perhaps the others in that framework as well) uses PMIx is incorrect. These are shared memory components, and so they should not be posting their modex information on the "global" scope. They should be putting it solely on PMIX_LOCAL so it stays on the node. They should then be looking solely at their local peers and performing PMIx_Get on the values from just those procs.