Investigate and document current behavior of "aggressive" mode

[qkoziol]

Is "aggressive" mode really determined by the slot count provided by PRRTE? Or is it determined by a query to hwloc with a reference to the number of processes per node. It just surprises me that this part of OMPI is controlled by PRRTE instead of something more generic that might work with, say, Slurm direct launch via srun. (from @jjhursey)

[qkoziol]

This is an old link to section 10.8.21 -- it may or may not be correct: https://ompi--8329.org.readthedocs.build/en/8329/faq/running-mpi-apps.html

[qkoziol]

Split out from #10480

[edgargabriel]

Here is what I found so far by looking into the ompi source code:

1. The mca parameter that controls the yield_when_idle behavior is mpi_yield_when_idle (registered in mpi/runtime/ompi_mpi_params.c)

2. The only location that I could find that overrides the value of the MCA parameter is in ompi/instance/instance.c https://github.com/open-mpi/ompi/blob/main/ompi/instance/instance.c#L415

   /* if we are oversubscribed, then set yield_when_idle
     * accordingly */
    if (ompi_mpi_oversubscribed) {
        ompi_mpi_yield_when_idle = true;
    }

This value is then used to set the opal_progress_yield_when_idle value https://github.com/open-mpi/ompi/blob/main/ompi/instance/instance.c#L765

    /* see if yield_when_idle was specified - if so, use it */
    opal_progress_set_yield_when_idle (ompi_mpi_yield_when_idle);

3. ompi_mpi_oversubscribed is set in ompi/runtime/ompi_rte.c https://github.com/open-mpi/ompi/blob/main/ompi/runtime/ompi_rte.c#L935

#ifdef PMIX_NODE_OVERSUBSCRIBED
    pname.jobid = opal_process_info.my_name.jobid;
    pname.vpid = OPAL_VPID_WILDCARD;
    OPAL_MODEX_RECV_VALUE_OPTIONAL(ret, PMIX_NODE_OVERSUBSCRIBED, &pname,
                                   NULL, PMIX_BOOL);
    if (PMIX_SUCCESS == ret) {
        ompi_mpi_oversubscribed = true;
    }
#endif

I could not find another place where either ompi_mpi_oversubscribed or ompi_mpi_yield_when_idle is changed (or opal_progress_yield_when_idle outside of opal_progress_set_yield_when_idle() ). I will try to understand the prrte part next, but I think it is fair to say that ompi does whatever prrte returns here.

[edgargabriel]

After looking into this a bit more here is how I understand the logic:

• the yield_when_idle behavior (aka "aggressive mode") is controlled by the ompi_mpi_oversubscribed flag or the mca parameter mpi_yield_when_idle

• the flag is set as a result of a PMIx_Get operation on PMIX_NODE_OVERSUBSCRIBED. In case of a direct launch, it is up the the PMIx implementation whether that flag is supported and provided. Hence, the flag could be provided e.g. by slurm.

• In case of PRRTE, the flag is set in src/mca/rmaps/base/rmaps_base_support_fns.c in prte_rmaps_base_check_oversubscribed(). More specifically, it compares the number of slots on a node to the number of processes assigned. The number of slots again might be retrieved from hwloc (prte_hwloc_base_get_nbobjs_by_type() ) but can also come from a variety of ras components or by reading a host file. I don't think it is necessary to detail this path to answer the question on this ticket and update the documentation though.

[edgargabriel]

I read through the documentation of v5.0.x, and I don't see the documentation making wrong statements wrt to oversubscription. In addition, I am honestly not sure whether the 'explanation' on why something is behaving a certain way (vs. the how the user can influence the setting) warrants a 'blocking' label.

Unless there are objections, I would like to i) remove the blocking label and change it to a lower level, and ii) close this ticket sometime next week

[jsquyres]

@edgargabriel If the current statements in the docs are correct, cool -- I agree: remove the blocker label. The issue was that there were so many things that had changed about run-time behavior that it warranted a check to ensure that the docs were a) correct, and/or b) should be expanded/clarified/whatever.

If we want to additionally expand the current docs (with more explanations, examples, ...etc.) and those aren't critical / could be added at any time after v5.0.0, cool -- we can do that, too.

[edgargabriel]

@jsquyres I agree. I am 99.9% sure that we don't make an outrageously wrong statement related to oversubscription and the 'aggresive' mode control. If nothing else, I would argue that this issue does not warrant holding up the 5.0. release. Improvements are always possible.