ompi_mpi_instance_init_common needs to be local

[hppritcha]

ompi_mpi_instance_init_common is invoked when an application uses either MPI_Init or MPI_Session_init. The later function has purely local semantics to the invoking application. As is though, this function has to have a PMIx fence involving all procs in a "job" as some PMLs require this for the add_procs method to work correctly, even if just the "local" procs are included in the argument array.

This call and the preceding PMIx Fence need to be moved into the MPI_Init specific code. Code needs to be added in the MPI_Comm_create_from_group and MPI_Intercomm_create_from_groups to do the add_procs at that point. This may require either new PML methods or modification to add_proc methods that don't check for previously added procs.

[hppritcha]

related to #11212

[rhc54]

@hppritcha Just wondering: I have been working on a side-branch to ensure that PMIx_Group_construct properly returns all the job-level info and modex info for all group participants. If that is correctly completed, then would that remove this problem? Just trying to understand why "add_procs" would require a fence operation if all the job-level and modex info were known.

[hppritcha]

i'm not sure that would help. i'd need to double check but i'm pretty sure one thing that messes things up is the way smsc tries to get itself set up using the IMMEDIATE attribute for a pmix_get operation.

[rhc54]

Precisely - immediate means "check the server and return not found if the server doesn't have it". Right now, the server will not have the info because the group construct isn't correctly returning it when completing the operation. Hence my question. If "add_procs" requires a fence for some reason other than needing endpt info, then it won't help. But if not finding info is the only reason it is failing, then fixing the group construct is the right answer.