rmaps: fixed the ordering of mpirun target nodes

[karasevb]

Fixed the desync of job-nodelists between mpirun and orted
daemons. The issue was observed when using RSH launching because user
can provide arbitrary order of nodes regarding HNP placement.
The mpirun process propagate the daemon's nodelist order to nodes.
The problem was that HNP itself is assembling the nodelist based on
user provided order. As the result ranks assignment was calculated
differently on orted and mpirun.

Consider following example:

• User launches mpirun on node cn2.
• Hostlist is cn1,cn2,cn3,cn4; ppn=1
• mpirun is passing hostlist cn[2:2,1,3-4]@0(4) to orteds
  So as result mpirun will assing rank 0 on cn1 while orted will assign
  rank 0 on cn2 (because orted sees cn2 as the first element in the node
  list)

Signed-off-by: Boris Karasev [email protected]

[karasevb]

[cn02:113267] [[23760,0],0] plm:rsh: final template argv:
        /bin/ssh <template>     PATH=/home/user/install/master/bin:$PATH ; export PATH ; LD_LIBRARY
_PATH=/home/user/install/master/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; DYLD_LIBRARY_PATH=/
labhome/boriska/scrap/2018_01_25_issue_4516/install/master/lib:$DYLD_LIBRARY_PATH ; export DYLD_LIBRARY_PATH ;   
/home/user/install/master/bin/orted -mca orte_debug_daemons "1" -mca ess "env" -mca ess_base_jobid "1557135360"
-mca ess_base_vpid "<template>" -mca ess_base_num_procs "4"

 -mca orte_node_regex "cn[2:2,1,3-4]@0(4)"     <-----

 -mca orte_hnp_uri "1557135360.0;tcp:
//192.168.0.1:42899" -mca orte_plm_base_verbose "10" -mca plm_base_verbose "10" -mca ess_base_verbos
e "0" -mca opal_pmix_base_verbose "0" -mca orte_odls_base_verbose "0" --mca orte_rmaps_base_verbose "0" -mca orte_ras_base_verbose "0
" -mca orte_nidmap_verbose "0" --mca btl "tcp,self" --mca btl_tcp_if_include "em1" -mca plm "rsh" --tree-spawn -mca orte_parent_uri "
1557135360.0;tcp://192.168.0.1:42899" -mca rmaps_ppr_pernode "1" -mca pmix "^s1,s2,cray,isolated"
...
[cn02:112049] mca:rmaps:base: assigning rank 0 to node cn01  <-----
[cn02:112049] mca:rmaps:base: assigning rank 1 to node cn02
[cn02:112049] mca:rmaps:base: assigning rank 2 to node cn03
[cn02:112049] mca:rmaps:base: assigning rank 3 to node cn04
...
[cn01:142926] mca:rmaps:base: assigning rank 0 to node cn02  <-----
[cn01:142926] mca:rmaps:base: assigning rank 1 to node cn01
[cn01:142926] mca:rmaps:base: assigning rank 2 to node cn03
[cn01:142926] mca:rmaps:base: assigning rank 3 to node cn04

[karasevb]

Fixes #4516

[karasevb]

@jladd-mlnx

[jladd-mlnx]

👍 Nicely done, Boris!

[jladd-mlnx]

@gpaulsen @jjhursey IBM, would you mind testing and confirming this fixes the issue?

[jjhursey]

yeah we can test, and confirm. Hopefully get back to you later today.

[sam6258]

The fix looks good. I reproduced the original issue with the master branch, and could not reproduce after applying PR.

[jjhursey]

Thanks @sam6258 !
@jladd-mlnx This looks good to go from us. Thanks for the fix.