There are not enough slots available in the system to satisfy the 1 slots

[cshanbo]

Hi all,
I tested a multi-node task with openmpi in this project.

As In the title, when I comment --oversubscribe in this line, there will be an exception, There are not enough slots available ... even when I only launch one task on each node.

I know it may raise this error when multiple tasks are launched, but is this normal of raising the exception when there's only one task?

Thank you

[rhc54]

It always helps us to properly respond if you tell us which version of OMPI you are using, whether this is a managed environment or not, and if not, then what was in your hostfile?

[cshanbo]

Thank you for your reply.

I'm using the latest version of openmpi on this page.
I launch the processes in a python script here. Basically, it's a bash command like
mpirun -np 2 -H host1,host2 ...

[ggouaillardet]

@cshanbo is mpirun invoked on one of the nodes listed in the -H option ?
what if you manually ?
mpirun -np 2 -H host1,host2 hostname
if it fails, what if you ?
mpirun -np 2 -H host1:1,host2:1 hostname
it do not think that should be necessary, but since there were quite some changes in this area, i might be wrong

[rhc54]

Adding the colon qualifiers isn't necessary. We automatically assign 1 slot for every time you list the host. My best guess is that there is something in your Python wrapper that is messing up the cmd line. Try printing out the actual cmd line you are executing, and then try invoking it manually to see what happens.

[cshanbo]

I tried to invoke the process from command line without python wrapper, still got the same error.
The command is simply like:

mpirun -mca mpi_warn_on_fork 0 -mca orte_base_help_aggregate 0 -np 2 -H nmyjs_104_22,nmyjs_104_37 -x PATH -x LD_LIBRARY_PATH -x PLATOON_TEST_WORKERS_NUM -x THEANO_FLAGS=floatX=float32,device=cpu /search/odin/chengshanbo/anaconda2/bin/python -u -m platoon.channel.controller test_multi_node PLATOON_LOGS/test_multi_node/2017-03-14_16-43-35 --multi

The THEANO_FLAGS, etc, parameters are needed by my codes.

Error

There are not enough slots available in the system to satisfy the 1 slots
that were requested by the application:
  /search/odin/chengshanbo/anaconda2/bin/python

Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 1 slots
that were requested by the application:
  /search/odin/chengshanbo/anaconda2/bin/python

Either request fewer slots for your application, or make more slots available
for use.

[rhc54]

Let's break the problem down into some smaller steps. First, let's get mpirun working correctly. Try running mpirun -np 2 -H <host1>,<host2> hostname

[zhangciba]

Hi every body.Is this problem solved?I meet the same problem.

[cshanbo]

Hi @zhangciba I instead using the --oversubscribe flag to "solve" the problem

[zhangciba]

I just submit the open-mpi jobs through Torque PBS system and I set the '#PBS -l nodes=32:ppn=16' parameter in the job script. And I also set the -np and -hostfile at the mpirun command. So I meet the problem. I remove the -np and -hostfile parameter in the mpirun command ,then the problem is solved.

[ggouaillardet]

You need to make sure Open MPI was built with tm support.
If you configure --with-tm, it will abort if there is no such support.

What if you simply

mpirun hostname

In your script ?
Do you see 512 tasks on 16 nodes ?
If you only see 16 tasks on the (first) node, that would strongly hint tm support was not built.

[zhangciba]

@ggouaillardet Thank you for your detailed answer.

I have checked my Open MPI and find it was configured with the 'tm' support.

Below is my job script named hostname.pbs

#!/bin/bash
#PBS -q medium
#PBS -l nodes=2:ppn=16
#PBS -N hostname

mpirun -np 32 -hostfile /home/zhangchi/study/jobs/hostfile hostname

the content of /home/zhangchi/study/jobs/hostfile

node233
node234

why I put node233 and node234 in hostfile ?
At that time the queue was not busy. And the free node list is like this

node233
node234
node235
...
...

So I think my job will be scheduled into these two nodes.

and the property of the queue medium was like this

#
# Create queues and set their attributes.
#
#
# Create and define queue medium
#
create queue medium
set queue medium queue_type = Execution
set queue medium max_queuable = 20
set queue medium max_user_queuable = 3
set queue medium max_running = 20
set queue medium acl_user_enable = True
set queue medium acl_users = ionization
set queue medium acl_users += zhangchi
set queue medium resources_max.ncpus = 16
set queue medium resources_max.nodes = 4
set queue medium resources_default.nodes = 1
set queue medium resources_default.walltime = 240:00:00
set queue medium max_user_run = 2
set queue medium enabled = True
set queue medium started = True

I submited this job using hostname.pbs and get the job number 5402, then the job completed soon.
And the job was actually running on node233 and node234
I checked the output file named hostname.o5402. It was like this.

[zhangchi@grid jobs]$ cat hostname.o5402
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 32 slots
that were requested by the application:
  hostname

Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------

Then I submit the same script the second time, I got different messages.
The job was running on node181 and node234.

[zhangchi@grid jobs]$ cat hostname.o5409
--------------------------------------------------------------------------
A hostfile was provided that contains at least one node not
present in the allocation:

  hostfile:  /home/zhangchi/study/jobs/hostfile
  node:      node233

If you are operating in a resource-managed environment, then only
nodes that are in the allocation can be used in the hostfile. You
may find relative node syntax to be a useful alternative to
specifying absolute node names see the orte_hosts man page for
further information.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
ORTE was unable to reliably start one or more daemons.
This usually is caused by:

* not finding the required libraries and/or binaries on
  one or more nodes. Please check your PATH and LD_LIBRARY_PATH
  settings, or configure OMPI with --enable-orterun-prefix-by-default

* lack of authority to execute on one or more specified nodes.
  Please verify your allocation and authorities.

* the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base).
  Please check with your sys admin to determine the correct location to use.

*  compilation of the orted with dynamic libraries when static are required
  (e.g., on Cray). Please check your configure cmd line and consider using
  one of the contrib/platform definitions for your system type.

* an inability to create a connection back to mpirun due to a
  lack of common network interfaces and/or no route found between
  them. Please check network connectivity (including firewalls
  and network routing requirements).
--------------------------------------------------------------------------

Because that was my first time using Torque PBS to submit jobs.So I did not know how to write the job script. After search the same question on the web. I corrected the script like this(just remove the -np and -hostfile option).

[zhangchi@grid jobs]$ cat hostname.pbs
#!/bin/bash
#PBS -q medium
#PBS -l nodes=2:ppn=16
#PBS -N hostname

mpirun hostname

[zhangchi@grid jobs]$ qsub hostname.pbs
5405.grid
[zhangchi@grid jobs]$ cat hostname.o5405
node236
node236
node236
node236
node236
node236
node236
node236
node236
node236
node236
node236
node236
node236
node236
node236
node234
node234
node234
node234
node234
node234
node234
node234
node234
node234
node234
node234
node234
node234
node234
node234

The job could work normally and I think that not only the real resource lack but also the wrong way of using job script will lead to the problem 'not enough slots available'.