setting mpi_yield_when_idle in etc/openmpi-mca-params.conf does not seem to work in 4.0.0

[iassiour]

Consider the following program where I force some processes to wait on a Bcast call

int main(int argc, char **argv) {


  int myrank, world_size;
  MPI_Init(&argc, &argv);
  MPI_Status * status;

//  opal_progress_set_event_flag(1);

  MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
  MPI_Comm_size(MPI_COMM_WORLD, &world_size);

  int istat;

  int name_len,i;
  char processor_name[MPI_MAX_PROCESSOR_NAME];
  MPI_Get_processor_name(processor_name, &name_len);

  printf("Hello world from processor %s, rank %d out of %d processors\n",
           processor_name, myrank, world_size);

  if(myrank == 0)
     while(1){};

  istat = MPI_Bcast(processor_name,name_len, MPI_CHAR, 0, MPI_COMM_WORLD);

  MPI_Finalize();
  return 0;
}

If I run this setting the -mca mpi_yield_when_idle true in mpirun

the processes seems to spend 85% in system and 15% in user, this is what I would have expected from yield option.

However, If I remove the -mca mpi_yield_when_idle from mpirun and set this through the etc/openmpi-mca-params.conf

( adding this line mpi_yield_when_idle = 1)

I see 100% time on user:

Could you please confirm that setting this option through the configuration file does work as expected in later openmpi versions.

[bosilca]

First, make sure the MCA file you are adding your options to is part of the mca_param_files list. Check this with ompi_info --param all all -l 9 | grep mca_param_files.

It seems to work just fine with 4.x and master:

$ ompi_info --param all all -l 9 | grep yield
            MCA mpi base: parameter "mpi_yield_when_idle" (current value: "true", data source: file (/Users/bosilca/opt/ompi/release/4.0.x/debug/etc/openmpi-mca-params.conf:60), level: 9 dev/all, type: bool)

[iassiour]

Hi bosilca,

I see that it correctly picking the configuration file:

ompi_info --param all all -l 9 | grep yield
MCA mpi base: parameter "mpi_yield_when_idle" (current value: "true", data source: file (/programming/ioannis/openmpi/4.0.0/etc/openmpi-mca-params.conf:67), level: 9 dev/all, type: bool)

However I still observe the above behavior. Any ideas?

[bosilca]

RTE is forcing the value of mpi_yield_when_idle via an environment variable according to the detected oversubscription. The order of the MCA variables being arguments, env, file, in the first case it works because the MCA variable is found in the arguments, and therefore the value set by RTE is ignored. However, as RTE is setting it in the environ, the value provided by the user in any configuration file is blatantly ignored.

This is a very confusing and certainly unexpected behavior. @rhc54 can certainly help fixing this.

[iassiour]

Hi bosilca,

Thanks, just to note that in both experiments I also set the -oversubscribe in mpirun command.
Given that shouldn't RTE force to mpi_yield_when_idle anyway? How it detects oversubscription?

[bosilca]

I would have expected RTE to force the yield_when_idle only when there was oversubscription (aka number of processes per node larger than the number of available resources). RTE could also use a second mechanism to suggest to yield so that users always make the final decision and have the opportunity to explicitly overwrite RTE decision.

[ggouaillardet]

Is this a duplicate of #6433 ?
I fixed it in #6440 but it seems it was never backported into the branch release

[jsquyres]

@ggouaillardet Could you make PRs for the relevant release branches?