v2.x: MPI singleton + PMIx dstore fails

[kawashima-fj]

@rhc54 As discussed in #2859, when I enable the PMIx dstore, An MPI process of singleton execution (launch directly; no mpiexec) fails with the following message on v2.x branch.

--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):
  
  orte_ess_init failed
  --> Returned value Bad parameter (-5) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):
  
  ompi_mpi_init: ompi_rte_init failed
  --> Returned "Bad parameter" (-5) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)

The problem will be in fork_hnp function of the singleton ESS. It checks the number of PMIx parameters. But the number varies if dstore is enabled. Probably PMIX_DSTORE_ESH_BASE_PATH is added.

https://github.com/open-mpi/ompi/blob/v2.x/orte/mca/ess/singleton/ess_singleton_module.c#L615

615             if (4 != opal_argv_count(argv)) {
(gdb) n
616                 opal_argv_free(argv);
(gdb) p cptr
$8 = 0x6533cc "PMIX_NAMESPACE=399310849,PMIX_RANK=0,PMIX_SERVER_URI=pmix-server:22133:/tmp/openmpi-sessions-1000@imtofu2_0/6093/pmix-22133,PMIX_SECURITY_MODE=native,PMIX_DSTORE_ESH_BASE_PATH=/tmp/openmpi-sessions-10"...

The master seems to have the solution. Probably a1e8e58. Cherry-picking this commit is sufficient?

[kawashima-fj]

At least 93e7384 and fb5bcc4 is also needed in addition to a1e8e58. I cannot determine other commits in orte/mca/ess/singleton in master is needed. @rhc54 @ggouaillardet Could you take a look?

[ggouaillardet]

@kawashima-fj at first glance, that looks good to me
fwiw, i just noted fb5bcc4 not only plugs a memory leak (as indicated by the commit message), but it also fixes an array overflow when more than 4 PMIX_* environment variables are set.
thanks !

[hppritcha]

This PR requires a reviewer.

[ggouaillardet]

@hppritcha this is an issue, not a PR
The referenced PR was both reviewed and merged, so i am closing this issue