Error launching under slurm (Out of resource)

[gkatev]

Hi, I've been unable to start mpi jobs under slurm reservations with the latest main.
I'm under salloc -N 1 -n 48, and the mesage is:

$ mpirun -n 1 hostname
--------------------------------------------------------------------------
Your job failed to map. Either no mapper was available, or none
of the available mappers was able to perform the requested
mapping operation.

  Mapper result:       Out of resource
  Application:         hostname
  #procs to be mapped: 1
  Mapping policy:      BYSLOT
  Binding policy:      CORE

--------------------------------------------------------------------------

This happens with 1 as well as with 2 nodes in the reservation. It also doesn't work in 5.0.x, but in 5.0.0rc8 all is well. It doesn't happen when not under slurm.

I tried to chase it down a bit:

$ mpirun -n 1 --prtemca rmaps_base_verbose 10 --display alloc --output tag hostname
[deepv:02593] mca: base: component_find: searching NULL for rmaps components
[deepv:02593] mca: base: find_dyn_components: checking NULL for rmaps components
[deepv:02593] pmix:mca: base: components_register: registering framework rmaps components
[deepv:02593] pmix:mca: base: components_register: found loaded component ppr
[deepv:02593] pmix:mca: base: components_register: component ppr register function successful
[deepv:02593] pmix:mca: base: components_register: found loaded component rank_file
[deepv:02593] pmix:mca: base: components_register: component rank_file has no register or open function
[deepv:02593] pmix:mca: base: components_register: found loaded component round_robin
[deepv:02593] pmix:mca: base: components_register: component round_robin register function successful
[deepv:02593] pmix:mca: base: components_register: found loaded component seq
[deepv:02593] pmix:mca: base: components_register: component seq register function successful
[deepv:02593] [prterun-deepv-2593@0,0] rmaps:base set policy with slot
[deepv:02593] mca: base: components_open: opening rmaps components
[deepv:02593] mca: base: components_open: found loaded component ppr
[deepv:02593] mca: base: components_open: component ppr open function successful
[deepv:02593] mca: base: components_open: found loaded component rank_file
[deepv:02593] mca: base: components_open: found loaded component round_robin
[deepv:02593] mca: base: components_open: component round_robin open function successful
[deepv:02593] mca: base: components_open: found loaded component seq
[deepv:02593] mca: base: components_open: component seq open function successful
[deepv:02593] mca:rmaps:select: checking available component ppr
[deepv:02593] mca:rmaps:select: Querying component [ppr]
[deepv:02593] mca:rmaps:select: checking available component rank_file
[deepv:02593] mca:rmaps:select: Querying component [rank_file]
[deepv:02593] mca:rmaps:select: checking available component round_robin
[deepv:02593] mca:rmaps:select: Querying component [round_robin]
[deepv:02593] mca:rmaps:select: checking available component seq
[deepv:02593] mca:rmaps:select: Querying component [seq]
[deepv:02593] [prterun-deepv-2593@0,0]: Final mapper priorities
[deepv:02593] 	Mapper: rank_file Priority: 100
[deepv:02593] 	Mapper: ppr Priority: 90
[deepv:02593] 	Mapper: seq Priority: 60
[deepv:02593] 	Mapper: round_robin Priority: 10

======================   ALLOCATED NODES   ======================
    dp-dam01: slots=48 max_slots=0 slots_inuse=0 state=UP
	Flags: DAEMON_LAUNCHED:SLOTS_GIVEN
	aliases: 10.2.10.41,10.2.17.81
=================================================================
[deepv:02593] mca:rmaps: mapping job prterun-deepv-2593@1
[deepv:02593] mca:rmaps: setting mapping policies for job prterun-deepv-2593@1 inherit TRUE hwtcpus FALSE
[deepv:02593] mca:rmaps mapping given by MCA param
[deepv:02593] mca:rmaps[540] default binding policy given
[deepv:02593] mca:rmaps:rf: job prterun-deepv-2593@1 not using rankfile policy
[deepv:02593] mca:rmaps:ppr: job prterun-deepv-2593@1 not using ppr mapper PPR NULL policy PPR NOTSET
[deepv:02593] [prterun-deepv-2593@0,0] rmaps:seq called on job prterun-deepv-2593@1
[deepv:02593] mca:rmaps:seq: job prterun-deepv-2593@1 not using seq mapper
[deepv:02593] mca:rmaps:rr: mapping job prterun-deepv-2593@1
[deepv:02593] [prterun-deepv-2593@0,0] Starting with 1 nodes in list
[deepv:02593] [prterun-deepv-2593@0,0] Filtering thru apps
[deepv:02593] [prterun-deepv-2593@0,0] Retained 1 nodes in list
[deepv:02593] [prterun-deepv-2593@0,0] node dp-dam01 has 48 slots available
[deepv:02593] AVAILABLE NODES FOR MAPPING:
[deepv:02593]     node: dp-dam01 daemon: 1 slots_available: 48
[deepv:02593] mca:rmaps:rr: mapping by slot for job prterun-deepv-2593@1 slots 48 num_procs 1
[deepv:02593] mca:rmaps:rr:slot working node dp-dam01
[deepv:02593] [prterun-deepv-2593@0,0] get_avail_ncpus: node dp-dam01 has 0 procs on it
[deepv:02593] mca:rmaps:rr:slot job prterun-deepv-2593@1 is oversubscribed - performing second pass
[deepv:02593] mca:rmaps:rr:slot working node dp-dam01
[deepv:02593] [prterun-deepv-2593@0,0] get_avail_ncpus: node dp-dam01 has 0 procs on it
--------------------------------------------------------------------------
Your job failed to map. Either no mapper was available, or none
of the available mappers was able to perform the requested
mapping operation.

  Mapper result:       Out of resource
  Application:         hostname
  #procs to be mapped: 1
  Mapping policy:      BYSLOT
  Binding policy:      CORE

--------------------------------------------------------------------------

It looked to me like the failure starts happening because prte_rmaps_base_get_ncpus() returned 0. These debug prints:

diff --git a/src/mca/rmaps/base/rmaps_base_support_fns.c b/src/mca/rmaps/base/rmaps_base_support_fns.c
index 8a2974a90f..c345c2e727 100644
--- a/src/mca/rmaps/base/rmaps_base_support_fns.c
+++ b/src/mca/rmaps/base/rmaps_base_support_fns.c
@@ -668,6 +668,7 @@ int prte_rmaps_base_get_ncpus(prte_node_t *node,
     int ncpus;
 
 #if HWLOC_API_VERSION < 0x20000
+    printf("HWLOC_API_VERSION < 0x20000\n");
     hwloc_obj_t root;
     root = hwloc_get_root_obj(node->topology->topo);
     if (NULL == options->job_cpuset) {
@@ -679,6 +680,7 @@ int prte_rmaps_base_get_ncpus(prte_node_t *node,
         hwloc_bitmap_and(prte_rmaps_base.available, prte_rmaps_base.available, obj->allowed_cpuset);
     }
 #else
+    printf("HWLOC_API_VERSION >= 0x20000\n");
     if (NULL == options->job_cpuset) {
         hwloc_bitmap_copy(prte_rmaps_base.available, hwloc_topology_get_allowed_cpuset(node->topology->topo));
     } else {
diff --git a/src/mca/rmaps/round_robin/rmaps_rr_mappers.c b/src/mca/rmaps/round_robin/rmaps_rr_mappers.c
index 484449ce7a..b3e631fea6 100644
--- a/src/mca/rmaps/round_robin/rmaps_rr_mappers.c
+++ b/src/mca/rmaps/round_robin/rmaps_rr_mappers.c
@@ -123,6 +123,7 @@ pass:
          * the user didn't specify a required binding, then we set
          * the binding policy to do-not-bind for this node */
         ncpus = prte_rmaps_base_get_ncpus(node, NULL, options);
+        printf("prte_rmaps_base_get_ncpus() = %d\n", ncpus);
         if (options->nprocs > ncpus &&
             options->nprocs <= node->slots_available &&
             !PRTE_BINDING_POLICY_IS_SET(jdata->map->binding)) {

Produce:

prte_rmaps_base_get_ncpus() = 0
HWLOC_API_VERSION >= 0x20000

[rhc54]

Try this while under the Slurm allocation: mpirun --prtemca plm ssh -n 1 hostname. Also, what version of Slurm are you using?

[gkatev]

Do we expect this command to hang? (it does). I'm using my cluster's system-wide version, which as I see is 22.05.7.

[gkatev]

I realized I had map-by=slot and bind-to=core set in an env var. Apparently without these params the problem does not appear. But I don't suppose this combination is "illegal"?

Revised effects without any env vars:

$ mpirun -n 1 --map-by slot --bind-to core hostname
--------------------------------------------------------------------------
Your job failed to map. Either no mapper was available, or none
of the available mappers was able to perform the requested
mapping operation.

  Mapper result:       Out of resource
  Application:         hostname
  #procs to be mapped: 1
  Mapping policy:      BYSLOT
  Binding policy:      CORE

--------------------------------------------------------------------------

$ mpirun -n 1 --bind-to core hostname
--------------------------------------------------------------------------
Your job failed to map. Either no mapper was available, or none
of the available mappers was able to perform the requested
mapping operation.

  Mapper result:       Out of resource
  Application:         hostname
  #procs to be mapped: 1
  Mapping policy:      BYCORE
  Binding policy:      CORE

--------------------------------------------------------------------------

$ mpirun -n 1 --map-by slot hostname
/* all good */

$ mpirun -n 1 hostname
/* all good */

Still stands that it appears in main and 5.0.x but not 5.0.0rc8.

Edit: When running the second command above (mpirun -n 1 --bind-to core hostname), that "HWLOC_API_VERSION >= 0x20000" debug print I had added appears in the output 2306 times!

[rhc54]

If you specify the binding, then we require that the binding be done - i.e., if you say --bind-to core, then we error out if we cannot do it for some reason. If you don't specify the binding, then the default binding becomes optional - so if we can't bind for some reason, we still go ahead and map (we just leave the resulting procs unbound).

My best guess here is that Slurm is assigning you to some place where we cannot bind your procs to cores. Perhaps there are only hwthreads and no cores? If you do --map-by slot:hwthreadcpus --bind-to hwthread, does it work?

I don't have access to a Slurm machine, but we aren't hearing of any problems from people who do - so I'm thinking there might be something about this Slurm setup that is causing the problem. I asked for the version because I was just contacted by SchedMD about a bug in one of their releases that causes OMPI some issues, but your version doesn't match so that isn't the cause.

[gkatev]

I see. Perhaps I also have to contact the system's maintentainers, I'm not super familiar with slurm either.

The hwthread command does yield an improvement, but I can only get at most 20 processes (seems arbitrary, system has 48/96 cores/threads across 2 sockets):

$ mpirun -n 1 --map-by slot:hwtcpus --bind-to hwthread hostname
HWLOC_API_VERSION >= 0x20000
prte_rmaps_base_get_ncpus() = 20
HWLOC_API_VERSION >= 0x20000
/* All good */

$ mpirun -n 21 --map-by slot:hwtcpus --bind-to hwthread hostname
HWLOC_API_VERSION >= 0x20000
prte_rmaps_base_get_ncpus() = 20
HWLOC_API_VERSION >= 0x20000
HWLOC_API_VERSION >= 0x20000
prte_rmaps_base_get_ncpus() = 0
HWLOC_API_VERSION >= 0x20000
/* Out of resource */

I will also try out a couple different salloc parameters (e.g. cpus instead of tasks, or something like that) to see if I spot an improvement.

[rhc54]

Could be some kind of cgroup setting - we are only seeing 20 hwthreads in the allocation. Sounds suspicious.

[naughtont3]

Not sure this fits all of your issue, but here are a few items we check/use for running prte/ompi within a local slurm system.

1. Configure/build

    ensure you configured with `--enable-slurm`

2. Ensure run using PLM and RAS as slurm modules

    export PRTE_MCA_ras=slurm
    export PRTE_MCA_plm=slurm

      # *** or directly on command-line ***
     
    mpirun \
        --prtemca plm slurm \
        --prtemca ras slurm \
        ...

3. Setting following options may also be helpful.

     # Useful to ensure can use all cores in allocation
     # when running mpirun in allocation
    export PRTE_MCA_ras_slurm_use_entire_allocation=1

     # Needed for VNI enabled Cray XE SS11 system
    export PRTE_MCA_ras_base_launch_orted_on_hn=1

      # *** or directly on command-line ***

    mpirun \
        --prtemca ras_slurm_use_entire_allocation 1 \
        --prtemca ras_base_launch_orted_on_hn 1 \
        ...

4. SLURM allocation option for reserving service cores, which will reduce the number of cores available for apps.

   The -S n option can control this, e.g., CRUSHER @ OLCF currently defaults to -S 8 (56 cores).

   salloc -S 0 ... # do not reserve any cores for system noise
   salloc -S 1 ... # reserve 1 core
   salloc -S 8 ... # reserve 8 cores (one per l3cache) [default]

Look at SLURM_JOB_CPUS_PER_NODE inside the allocation to better understand what you have available. This value is used for #slots when using the ras_slurm_use_entire_allocation=1 option

5. SLURM allocation option for hwthreads mapping/usage, if intend to use hwthreads for apps (e.g., --map-by hwthread)
   then you will want to add --threads-per-core to your allocation request. For example,

     salloc -N 2 -t 10 -A $MYACCT -S 0 --threads-per-core 2

6. Useful debug options

  --prtemca ras_base_verbose 20           # See that #slots matches SLURM_JOB_CPUS_PER_NODE
  --prtemca plm_base_verbose 20           # Ensure slurm selected

[gkatev]

Regarding the initial claim about the versions, there was a mistake in my 5.0.0rc8 installation. So after clearing that up, it is in fact 5.0.0rc7 that works fine.

5.0.0rc8 also erros out, even without the bind-to param (maybe the default there was to bind to core?), but with a different message

$ 5.0.0.rc8/mpirun -n 1 hostname
--------------------------------------------------------------------------
All nodes which are allocated for this job are already filled.
--------------------------------------------------------------------------

$ 5.0.0.rc8/mpirun -n 1 --map-by slot hostname
--------------------------------------------------------------------------
All nodes which are allocated for this job are already filled.
--------------------------------------------------------------------------

$ 5.0.0.rc8/mpirun -n 1 --bind-to none hostname
--------------------------------------------------------------------------
All nodes which are allocated for this job are already filled.
--------------------------------------------------------------------------

5.0.x and main also error out, as described above (no change)

Do we know how this variable (5.0.0rc7 vs 5.0.0rc8) might play a role in this? I see a bunch of changes in prte_rmaps_rr_byslot (assuming this indeed used in these scenarios) between the versions in rc7 and rc8.

[rhc54]

I know that those release candidates are quite stale (in terms of PRRTE), and so I can't be sure you aren't just hitting old problems that have already been fixed. Likewise, OMPI main was just updated yesterday (IIRC - it was in the last day or two).

Are you using the head of OMPI main as of today? If not, it might be worth updating so we know we are all looking at the same thing. You might also want to go into the 3rd-party openpmix and prrte submodule directories and do "git checkout master" followed by "git pull" on each of them, just to ensure you have the latest of both code bases.

Please be sure to configure with --enable-debug.

The error message is really suspicious to me. For whatever reason, you don't seem to have a usable allocation. Let's try with an updated OMPI main (as per above) and see what you get. If it fails, then add "--prtemca rmaps_base_verbose 100" to the mpirun cmd line.

[gkatev]

I see. In the above I had prrte @ dc6ccf6 and openmpix @ 415d704 (few days old). I made a new build with the latest prrte/openmpix master.

I'm now at pprrte @ 081890a, openpmix @ 0818181 (debug build), the problem does remain.

$ mpirun -n 1 --map-by slot --bind-to core --prtemca rmaps_base_verbose 100 hostname
[deepv:22377] mca: base: component_find: searching NULL for rmaps components
[deepv:22377] mca: base: find_dyn_components: checking NULL for rmaps components
[deepv:22377] pmix:mca: base: components_register: registering framework rmaps components
[deepv:22377] pmix:mca: base: components_register: found loaded component ppr
[deepv:22377] pmix:mca: base: components_register: component ppr register function successful
[deepv:22377] pmix:mca: base: components_register: found loaded component rank_file
[deepv:22377] pmix:mca: base: components_register: component rank_file has no register or open function
[deepv:22377] pmix:mca: base: components_register: found loaded component round_robin
[deepv:22377] pmix:mca: base: components_register: component round_robin register function successful
[deepv:22377] pmix:mca: base: components_register: found loaded component seq
[deepv:22377] pmix:mca: base: components_register: component seq register function successful
[deepv:22377] mca: base: components_open: opening rmaps components
[deepv:22377] mca: base: components_open: found loaded component ppr
[deepv:22377] mca: base: components_open: component ppr open function successful
[deepv:22377] mca: base: components_open: found loaded component rank_file
[deepv:22377] mca: base: components_open: found loaded component round_robin
[deepv:22377] mca: base: components_open: component round_robin open function successful
[deepv:22377] mca: base: components_open: found loaded component seq
[deepv:22377] mca: base: components_open: component seq open function successful
[deepv:22377] mca:rmaps:select: checking available component ppr
[deepv:22377] mca:rmaps:select: Querying component [ppr]
[deepv:22377] mca:rmaps:select: checking available component rank_file
[deepv:22377] mca:rmaps:select: Querying component [rank_file]
[deepv:22377] mca:rmaps:select: checking available component round_robin
[deepv:22377] mca:rmaps:select: Querying component [round_robin]
[deepv:22377] mca:rmaps:select: checking available component seq
[deepv:22377] mca:rmaps:select: Querying component [seq]
[deepv:22377] [prterun-deepv-22377@0,0]: Final mapper priorities
[deepv:22377] 	Mapper: rank_file Priority: 100
[deepv:22377] 	Mapper: ppr Priority: 90
[deepv:22377] 	Mapper: seq Priority: 60
[deepv:22377] 	Mapper: round_robin Priority: 10
[deepv:22377] [prterun-deepv-22377@0,0] rmaps:base set policy with slot
[deepv:22377] mca:rmaps: mapping job prterun-deepv-22377@1
[deepv:22377] mca:rmaps: setting mapping policies for job prterun-deepv-22377@1 inherit TRUE hwtcpus FALSE
[deepv:22377] mca:rmaps:rf: job prterun-deepv-22377@1 not using rankfile policy
[deepv:22377] mca:rmaps:ppr: job prterun-deepv-22377@1 not using ppr mapper PPR NULL policy PPR NOTSET
[deepv:22377] [prterun-deepv-22377@0,0] rmaps:seq called on job prterun-deepv-22377@1
[deepv:22377] mca:rmaps:seq: job prterun-deepv-22377@1 not using seq mapper
[deepv:22377] mca:rmaps:rr: mapping job prterun-deepv-22377@1
[deepv:22377] [prterun-deepv-22377@0,0] Starting with 1 nodes in list
[deepv:22377] [prterun-deepv-22377@0,0] Filtering thru apps
[deepv:22377] [prterun-deepv-22377@0,0] Retained 1 nodes in list
[deepv:22377] [prterun-deepv-22377@0,0] node dp-dam01 has 48 slots available
[deepv:22377] AVAILABLE NODES FOR MAPPING:
[deepv:22377]     node: dp-dam01 daemon: 1 slots_available: 48
[deepv:22377] mca:rmaps:rr: mapping by slot for job prterun-deepv-22377@1 slots 48 num_procs 1
[deepv:22377] mca:rmaps:rr:slot working node dp-dam01
[deepv:22377] [prterun-deepv-22377@0,0] get_avail_ncpus: node dp-dam01 has 0 procs on it
[deepv:22377] mca:rmaps:rr:slot job prterun-deepv-22377@1 is oversubscribed - performing second pass
[deepv:22377] mca:rmaps:rr:slot working node dp-dam01
[deepv:22377] [prterun-deepv-22377@0,0] get_avail_ncpus: node dp-dam01 has 0 procs on it
--------------------------------------------------------------------------
Your job failed to map. Either no mapper was available, or none
of the available mappers was able to perform the requested
mapping operation.

  Mapper result:       Out of resource
  Application:         hostname
  #procs to be mapped: 1
  Mapping policy:      BYSLOT
  Binding policy:      CORE

--------------------------------------------------------------------------

Again with the hwthreads I can spawn up to 20 procs but not more. Could you elaborate a bit on the cgroup thing with which I don't have experience? Can I easily check it's not in effect?

Also thanks @naughtont3 for the debug info, generally things seem in order. Under my salloc -N 1 -n 48, which I hope is right (it's what I've used for some time now):

$ echo $SLURM_JOB_CPUS_PER_NODE 
96
$ echo $SLURM_NTASKS
48
$ echo $SLURM_NPROCS
48

Do we think this looks like a slurm installation issue or like a prrte issue? Should prte_rmaps_base_get_ncpus return 0? Is prte_rmaps_base_get_ncpus affected by slurm or does it just use hwloc to find the number of cpus on the node?

[rhc54]

Do we think this looks like a slurm installation issue or like a prrte issue?

Really not sure at this point. Your Slurm envars look correct, but it is also clear that we are not seeing the allocation.

Should prte_rmaps_base_get_ncpus return 0? Is prte_rmaps_base_get_ncpus affected by slurm or does it just use hwloc to find the number of cpus on the node?

If "get_ncpus" returns 0, then we can't assign a process to that location as we don't have any available cpus. It uses hwloc to find the cpus, but the number of available cpus is impacted by Slurm, which can control what hwloc sees by setting a "window" of allowed cpus (cgroups is the mechanism by which that is done).

Basically, think of it as Slurm "binding" anything you run to a specified set of cpus. mpirun can assign a process to any cpu inside that window, but not outside it.

In this case, the allocation output from mpirun indicates that it sees 48 slots allocated to it, which matches the SLURM_NTASKS we were provided. What we aren't seeing are the cpus on the node.

Try applying the following patch to the 3rd-party/prrte directory:

diff --git a/src/mca/rmaps/base/rmaps_base_support_fns.c b/src/mca/rmaps/base/rmaps_base_support_fns.c
index c7d04044ab..2fa6b7d6bc 100644
--- a/src/mca/rmaps/base/rmaps_base_support_fns.c
+++ b/src/mca/rmaps/base/rmaps_base_support_fns.c
@@ -674,6 +674,7 @@ int prte_rmaps_base_get_ncpus(prte_node_t *node,
                               prte_rmaps_options_t *options)
 {
     int ncpus;
+    char *tmp;
 
 #if HWLOC_API_VERSION < 0x20000
     hwloc_obj_t root;
@@ -687,15 +688,30 @@ int prte_rmaps_base_get_ncpus(prte_node_t *node,
         hwloc_bitmap_and(prte_rmaps_base.available, prte_rmaps_base.available, obj->allowed_cpuset);
     }
 #else
+    if (NULL != options->job_cpuset) {
+        tmp = NULL;
+        hwloc_bitmap_list_asprintf(&tmp, options->job_cpuset);
+        pmix_output(0, "JOBCPUSET: %s", (NULL == tmp) ? "NO-CPUS" : tmp);
+    } else {
+        pmix_output(0, "JOBCPUSET IS NULL");
+    }
+    tmp = NULL;
+    hwloc_bitmap_list_asprintf(&tmp, hwloc_topology_get_allowed_cpuset(node->topology->topo));
+    pmix_output(0, "ALLOWEDCPUSET: %s", (NULL == tmp) ? "NULL" : tmp);
     if (NULL == options->job_cpuset) {
         hwloc_bitmap_copy(prte_rmaps_base.available, hwloc_topology_get_allowed_cpuset(node->topology->topo));
     } else {
         hwloc_bitmap_and(prte_rmaps_base.available, hwloc_topology_get_allowed_cpuset(node->topology->topo), options->job_cpuset);
     }
+    pmix_output(0, "OBJ: %s", (NULL == obj) ? "NULL" : "NON-NULL");
     if (NULL != obj) {
         hwloc_bitmap_and(prte_rmaps_base.available, prte_rmaps_base.available, obj->cpuset);
     }
 #endif
+    tmp = NULL;
+    hwloc_bitmap_list_asprintf(&tmp, prte_rmaps_base.available);
+    pmix_output(0, "GETNCPUS: %s", (NULL == tmp) ? "NULL" : tmp);
+
     if (options->use_hwthreads) {
         ncpus = hwloc_bitmap_weight(prte_rmaps_base.available);
     } else {

Should hopefully provide a little more insight into what is going on.

[gkatev]

I see sounds reasonable. Let me also note that I'm not 100% confident in my salloc params, but I did also try other ones, e.g. with -t or -c instead of -n, but no dice.

With the above patch applied:

$ mpirun -n 1 --map-by slot --bind-to core --prtemca rmaps_base_verbose 100 hostname
[deepv:27740] mca: base: component_find: searching NULL for rmaps components
[deepv:27740] mca: base: find_dyn_components: checking NULL for rmaps components
[deepv:27740] pmix:mca: base: components_register: registering framework rmaps components
[deepv:27740] pmix:mca: base: components_register: found loaded component ppr
[deepv:27740] pmix:mca: base: components_register: component ppr register function successful
[deepv:27740] pmix:mca: base: components_register: found loaded component rank_file
[deepv:27740] pmix:mca: base: components_register: component rank_file has no register or open function
[deepv:27740] pmix:mca: base: components_register: found loaded component round_robin
[deepv:27740] pmix:mca: base: components_register: component round_robin register function successful
[deepv:27740] pmix:mca: base: components_register: found loaded component seq
[deepv:27740] pmix:mca: base: components_register: component seq register function successful
[deepv:27740] mca: base: components_open: opening rmaps components
[deepv:27740] mca: base: components_open: found loaded component ppr
[deepv:27740] mca: base: components_open: component ppr open function successful
[deepv:27740] mca: base: components_open: found loaded component rank_file
[deepv:27740] mca: base: components_open: found loaded component round_robin
[deepv:27740] mca: base: components_open: component round_robin open function successful
[deepv:27740] mca: base: components_open: found loaded component seq
[deepv:27740] mca: base: components_open: component seq open function successful
[deepv:27740] mca:rmaps:select: checking available component ppr
[deepv:27740] mca:rmaps:select: Querying component [ppr]
[deepv:27740] mca:rmaps:select: checking available component rank_file
[deepv:27740] mca:rmaps:select: Querying component [rank_file]
[deepv:27740] mca:rmaps:select: checking available component round_robin
[deepv:27740] mca:rmaps:select: Querying component [round_robin]
[deepv:27740] mca:rmaps:select: checking available component seq
[deepv:27740] mca:rmaps:select: Querying component [seq]
[deepv:27740] [prterun-deepv-27740@0,0]: Final mapper priorities
[deepv:27740] 	Mapper: rank_file Priority: 100
[deepv:27740] 	Mapper: ppr Priority: 90
[deepv:27740] 	Mapper: seq Priority: 60
[deepv:27740] 	Mapper: round_robin Priority: 10
[deepv:27740] [prterun-deepv-27740@0,0] rmaps:base set policy with slot
[deepv:27740] mca:rmaps: mapping job prterun-deepv-27740@1
[deepv:27740] mca:rmaps: setting mapping policies for job prterun-deepv-27740@1 inherit TRUE hwtcpus FALSE
[deepv:27740] mca:rmaps:rf: job prterun-deepv-27740@1 not using rankfile policy
[deepv:27740] mca:rmaps:ppr: job prterun-deepv-27740@1 not using ppr mapper PPR NULL policy PPR NOTSET
[deepv:27740] [prterun-deepv-27740@0,0] rmaps:seq called on job prterun-deepv-27740@1
[deepv:27740] mca:rmaps:seq: job prterun-deepv-27740@1 not using seq mapper
[deepv:27740] mca:rmaps:rr: mapping job prterun-deepv-27740@1
[deepv:27740] [prterun-deepv-27740@0,0] Starting with 1 nodes in list
[deepv:27740] [prterun-deepv-27740@0,0] Filtering thru apps
[deepv:27740] [prterun-deepv-27740@0,0] Retained 1 nodes in list
[deepv:27740] [prterun-deepv-27740@0,0] node dp-dam01 has 48 slots available
[deepv:27740] AVAILABLE NODES FOR MAPPING:
[deepv:27740]     node: dp-dam01 daemon: 1 slots_available: 48
[deepv:27740] mca:rmaps:rr: mapping by slot for job prterun-deepv-27740@1 slots 48 num_procs 1
[deepv:27740] mca:rmaps:rr:slot working node dp-dam01
[deepv:27740] JOBCPUSET: 0-19
[deepv:27740] ALLOWEDCPUSET: 0-95
[deepv:27740] OBJ: NULL
[deepv:27740] GETNCPUS: 0-19
prte_rmaps_base_get_ncpus() = 0
[deepv:27740] [prterun-deepv-27740@0,0] get_avail_ncpus: node dp-dam01 has 0 procs on it
[deepv:27740] JOBCPUSET: 0-19
[deepv:27740] ALLOWEDCPUSET: 0-95
[deepv:27740] OBJ: NULL
[deepv:27740] GETNCPUS: 0-19
[deepv:27740] mca:rmaps:rr:slot job prterun-deepv-27740@1 is oversubscribed - performing second pass
[deepv:27740] mca:rmaps:rr:slot working node dp-dam01
[deepv:27740] JOBCPUSET: 0-19
[deepv:27740] ALLOWEDCPUSET: 0-95
[deepv:27740] OBJ: NULL
[deepv:27740] GETNCPUS: 0-19
prte_rmaps_base_get_ncpus() = 0
[deepv:27740] [prterun-deepv-27740@0,0] get_avail_ncpus: node dp-dam01 has 0 procs on it
[deepv:27740] JOBCPUSET: 0-19
[deepv:27740] ALLOWEDCPUSET: 0-95
[deepv:27740] OBJ: NULL
[deepv:27740] GETNCPUS: 0-19
--------------------------------------------------------------------------
Your job failed to map. Either no mapper was available, or none
of the available mappers was able to perform the requested
mapping operation.

  Mapper result:       Out of resource
  Application:         hostname
  #procs to be mapped: 1
  Mapping policy:      BYSLOT
  Binding policy:      CORE

--------------------------------------------------------------------------