Skip to content

Device vendor ID: 0x02c9, part ID: 4123, WARNING: No preset parameters were found for the device that Open MPI detected #10841

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
LinuxPersonEC opened this issue Sep 23, 2022 · 17 comments
Open

Device vendor ID: 0x02c9, part ID: 4123, WARNING: No preset parameters were found for the device that Open MPI detected #10841

LinuxPersonEC opened this issue Sep 23, 2022 · 17 comments
Labels
question Target: v4.1.x

Comments

@LinuxPersonEC
Copy link

OpenMPI 4.1.1 in RHEL 8, 5e:00.0 Infiniband controller [0207]: Mellanox Technologies MT28908 Family [ConnectX-6] [15b3:101b]

Using this STREAM benchmark here are some verbose logs:

mpirun -mca pml_base_verbose 100 -mca btl_base_verbose 100 -mca mca_base_verbose 100 -mca btl_openib_verbose true  -mca pml ucx --mca orte_base_help_aggregate 0 --mca opal_warn_on_missing_libcuda 0 -np 1 --oversubscribe ./stream_mpi --oversubscribe
[g183:270526] mca: base: components_register: registering framework btl components
[g183:270526] mca: base: components_register: found loaded component openib
[g183:270526] mca: base: components_register: component openib register function successful
[g183:270526] mca: base: components_register: found loaded component sm
[g183:270526] mca: base: components_register: found loaded component tcp
[g183:270526] mca: base: components_register: component tcp register function successful
[g183:270526] mca: base: components_register: found loaded component self
[g183:270526] mca: base: components_register: component self register function successful
[g183:270526] mca: base: components_register: found loaded component vader
[g183:270526] mca: base: components_register: component vader register function successful
[g183:270526] mca: base: components_register: found loaded component smcuda
[g183:270526] mca: base: components_register: component smcuda register function successful
[g183:270526] mca: base: components_open: opening btl components
[g183:270526] mca: base: components_open: found loaded component openib
[g183:270526] mca: base: components_open: component openib open function successful
[g183:270526] mca: base: components_open: found loaded component tcp
[g183:270526] mca: base: components_open: component tcp open function successful
[g183:270526] mca: base: components_open: found loaded component self
[g183:270526] mca: base: components_open: component self open function successful
[g183:270526] mca: base: components_open: found loaded component vader
[g183:270526] mca: base: components_open: component vader open function successful
[g183:270526] mca: base: components_open: found loaded component smcuda
[g183:270526] btl: smcuda: cuda_max_send_size=131072, max_send_size=32768, max_frag_size=131072
[g183:270526] mca: base: components_open: component smcuda open function successful
[g183:270526] select: initializing btl component openib
[g183:270526] Checking distance from this process to device=mlx5_0
[g183:270526] hwloc_distances->nbobjs=4
[g183:270526] hwloc_distances->values[0]=10
[g183:270526] hwloc_distances->values[1]=21
[g183:270526] hwloc_distances->values[2]=11
[g183:270526] hwloc_distances->values[3]=21
[g183:270526] ibv_obj->type set to NULL
[g183:270526] Process is bound: distance to device is 0.000000
[g183][[11854,1],0][btl_openib_ini.c:172:opal_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 4123
[g183][[11854,1],0][btl_openib_ini.c:188:opal_btl_openib_ini_query] Found corresponding INI values: Mellanox ConnectX6
[g183][[11854,1],0][btl_openib_ini.c:172:opal_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0
[g183][[11854,1],0][btl_openib_ini.c:188:opal_btl_openib_ini_query] Found corresponding INI values: default
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   xxxx
  Local device: mlx5_0
--------------------------------------------------------------------------
[g183:270526] select: init of component openib returned failure
[g183:270526] mca: base: close: component openib closed
[g183:270526] mca: base: close: unloading component openib
[g183:270526] select: initializing btl component tcp
[g183:270526] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[g183:270526] btl: tcp: Found match: 127.0.0.1 (lo)
[g183:270526] btl:tcp: Attempting to bind to AF_INET port 1024
[g183:270526] btl:tcp: Successfully bound to AF_INET port 1024
[g183:270526] btl:tcp: my listening v4 socket is 0.0.0.0:1024
[g183:270526] btl:tcp: examining interface eth0
[g183:270526] btl:tcp: using ipv6 interface eth0
[g183:270526] btl:tcp: examining interface ib0
[g183:270526] btl:tcp: using ipv6 interface ib0
[g183:270526] select: init of component tcp returned success
[g183:270526] select: initializing btl component self
[g183:270526] select: init of component self returned success
[g183:270526] select: initializing btl component vader
[g183:270526] select: init of component vader returned failure
[g183:270526] mca: base: close: component vader closed
[g183:270526] mca: base: close: unloading component vader
[g183:270526] select: initializing btl component smcuda
[g183:270526] select: init of component smcuda returned failure
[g183:270526] mca: base: close: component smcuda closed
[g183:270526] mca: base: close: unloading component smcuda
[g183:270526] mca: base: components_register: registering framework pml components
[g183:270526] mca: base: components_register: found loaded component ucx
[g183:270526] mca: base: components_register: component ucx register function successful
[g183:270526] mca: base: components_open: opening pml components
[g183:270526] mca: base: components_open: found loaded component ucx
[g183:270526] mca: base: components_open: component ucx open function successful
[g183:270526] select: initializing pml component ucx
[g183:270526] select: init returned priority 51
[g183:270526] selected ucx best priority 51
[g183:270526] select: component ucx selected
-------------------------------------------------------------
STREAM version $Revision: 1.8 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Total Aggregate Array size = 105000000 (elements)
Total Aggregate Memory per array = 801.1 MiB (= 0.8 GiB).
Total Aggregate memory required = 2403.3 MiB (= 2.3 GiB).
Data is distributed across 1 MPI ranks
   Array size per MPI rank = 105000000 (elements)
   Memory per array per MPI rank = 801.1 MiB (= 0.8 GiB).
   Total memory per MPI rank = 2403.3 MiB (= 2.3 GiB).
-------------------------------------------------------------
Each kernel will be executed 10 times.
 The *best* time for each kernel (excluding the first iteration)
 will be used to compute the reported bandwidth.
The SCALAR value used for this run is 0.420000
-------------------------------------------------------------
Your timer granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 68148 microseconds.
   (= 68148 timer ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 timer ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:          13648.7     0.123378     0.123089     0.123952
Scale:         13784.7     0.122519     0.121874     0.123266
Add:           14363.8     0.175696     0.175440     0.175882
Triad:         14216.1     0.177668     0.177264     0.178539
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------
[g183:270526] mca: base: close: component ucx closed
[g183:270526] mca: base: close: unloading component ucx
[g183:270526] mca: base: close: component tcp closed
[g183:270526] mca: base: close: unloading component tcp
[g183:270526] mca: base: close: component self closed
[g183:270526] mca: base: close: unloading component self

WARNING: No preset parameters were found for the device that Open MPI detected:

  Local host:            xxxx
  Device name:           mlx5_0
  Device vendor ID:      0x02c9
  Device vendor part ID: 4123

I found a reference to this in the comments for mca-btl-openib-device-params.ini

# Note: Several vendors resell Mellanox hardware and put their own firmware
# on the cards, therefore overriding the default Mellanox vendor ID.
#
#     Mellanox      0x02c9

Forcing ucx still generates the warning:

mpirun  -mca pml ucx --mca orte_base_help_aggregate 0 --mca opal_warn_on_missing_libcuda 0 -np 1 --oversubscribe ./stream_mpi --oversubscribe
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   xxxx
  Local device: mlx5_0
--------------------------------------------------------------------------
-------------------------------------------------------------
STREAM version $Revision: 1.8 $

With version 4.1.1, if I use --mca btl 'openib' I get seg faults which I believe is expected as it's deprecated. I've tried --mca btl '^openib', --mca btl 'tcp' (or --mca btl 'tcp,self' using the OSU BMs) and the benchmark results are very similar even when I use multiple CPUs, threads and/or nodes. They also run without the warning messages. If I don't use a --mca option, I get the WARNING: message.

Does anyone know of a tried and true way to run these benchmarks so know if these MCA parameters make a difference or am I just not understanding how to use these? Perhaps running these benchmarks on a very active cluster with shared CPUs/nodes will affect the results?
@jsquyres
Copy link
Member

UCX is definitely the way to go; this is NVIDIA's preferred mechanism.

Sidenote: running with --mca btl openib is expected to fail, because you also need the self and vader BTLs for self-process-loopback and on-node-process communication, respectively. That being said, it seems a bit anti-social to segv instead of printing a helpful error message in these cases. ☹️

The warning message is definitely coming from the openib BTL (I just double checked that by checking for that message in the v4.1.x source -- it's only in the openib BTL).

I think you should be able to run with:

$ mpirun --mca pml ucx --mca btl '^openib' ...

This tells Open MPI to use the UCX PML for point-to-point MPI communication, and to skip using the openib BTL in any other contexts that it might come up (E.g., one sided MPI communication).

Any further suggestions from the @open-mpi/ucx team?

@LinuxPersonEC
Copy link
Author

That being said, it seems a bit anti-social to segv instead of printing a helpful error message in these cases. ☹️

Here's what it looks like running mpirun --mca btl 'openib' -np 4 --host xxx,yyy --oversubscribe --mca orte_base_help_aggregate 0 --mca opal_warn_on_missing_libcuda 0 ./stream_mpi

ucx_cuda_11.0.3_support/lib/libucs.so.0(+0x2b138) [0x155541ff9138]
=================================
/lib64/libpthread.so.0(+0x12dd0)[0x155554df7dd0]
[g123:119486] *** End of error message ***
[g123:119485] *** Process received signal ***
[g123:119485] Signal: Segmentation fault (11)
[g123:119485] Signal code:  (-6)
[g123:119485] Failing at address: 0x859d90001d2bd
==== backtrace (tid: 119487) ====
 0  /burg/opt/ucx_cuda_11.0.3_support/lib/libucs.so.0(ucs_handle_error+0x254) [0x155541ff8cd4]
 1  /burg/opt/ucx_cuda_11.0.3_support/lib/libucs.so.0(+0x2ae8c) [0x155541ff8e8c]
 2  /burg/opt/ucx_cuda_11.0.3_support/lib/libucs.so.0(+0x2b138) [0x155541ff9138]
=================================
[g123:119485] [ 0] /lib64/libpthread.so.0(+0x12dd0)[0x155554df7dd0]
[g123:119485] *** End of error message ***
[g123:119487] *** Process received signal ***
[g123:119487] Signal: Segmentation fault (11)
[g123:119487] Signal code:  (-6)
[g123:119487] Failing at address: 0x859d90001d2bf
[g123:119487] [ 0] /lib64/libpthread.so.0(+0x12dd0)[0x155554df7dd0]
[g123:119487] *** End of error message ***
--------------------------------------------------------------------------

And perhaps of interest the warning is clearer and no segv when running:

mpirun --mca btl 'openib' --mca pml ucx  -np 4 --host g123,g124  --oversubscribe --mca orte_base_help_aggregate 0 --mca opal_warn_on_missing_libcuda 0  ./stream_mpi

WARNING: There was an error initializing an OpenFabrics device.

  Local host:   xxxx
  Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
No components were able to be opened in the btl framework.

This typically means that either no components of this type were installed, or none of the installed components can be loaded. Sometimes this means that shared libraries required by these components are unable to be found/loaded.

Is there an option/parameter to verify ucx is available?

@LinuxPersonEC
Copy link
Author

Also when running mpirun --mca pml ucx --mca btl '^openib' on the OSU Multi-threaded Latency Test why can't ucx be selected?

[xxx:105832] PML ucx cannot be selected
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.

This typically means that either no components of this type were installed, or none of the installed components can be loaded. Sometimes this means that shared libraries required by these components are unable to be found/loaded.

  Host:      xxx
  Framework: pml
--------------------------------------------------------------------------
[xxx:2328334] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2198

@janjust
Copy link
Contributor

janjust commented Sep 23, 2022 via email

@LinuxPersonEC
Copy link
Author

UCX built with multithreaded support?

Ah good question was done before my time. Is there a way to use -x UCX_LOG_LEVEL=data to find out?

@jsquyres
Copy link
Member

I have a dim recollection that there were some bugs with UCX/UCT in the past that caused segvs when used in specific combinations. Do you have the most recent version of UCX?

@yosefe
Copy link
Contributor

yosefe commented Sep 25, 2022

@RobbieTheK can you pls run openmpi with "-mca pml_ucx_verbose 999" to show why UCX is not selected?

@LinuxPersonEC
Copy link
Author 

mpirun -mca pml_ucx_verbose 999 -np 6 --host xxx,xxx,xxx,xxx  --oversubscribe --mca orte_base_help_aggregate 0 --mca opal_warn_on_missing_libcuda 0  ./stream_mpi
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   xxx
  Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   xx
  Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   xxx
  Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   xxx
  Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   xxxx
  Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   xxx
  Local device: mlx5_0
--------------------------------------------------------------------------
[g090:1261043] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g090:1261044] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g091:1549665] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g091:1549666] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g093:1949867] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g092:1207223] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g091:1549665] common_ucx.c:304 posix/memory: did not match transport list
[g091:1549665] common_ucx.c:304 sysv/memory: did not match transport list
[g091:1549665] common_ucx.c:304 self/memory0: did not match transport list
[g091:1549665] common_ucx.c:304 tcp/ib0: did not match transport list
[g091:1549665] common_ucx.c:304 tcp/lo: did not match transport list
[g091:1549665] common_ucx.c:304 tcp/eth0: did not match transport list
[g090:1261043] common_ucx.c:304 posix/memory: did not match transport list
[g090:1261043] common_ucx.c:304 sysv/memory: did not match transport list
[g090:1261043] common_ucx.c:304 self/memory0: did not match transport list
[g090:1261043] common_ucx.c:304 tcp/lo: did not match transport list
[g090:1261043] common_ucx.c:304 tcp/eth0: did not match transport list
[g090:1261044] common_ucx.c:304 posix/memory: did not match transport list
[g090:1261044] common_ucx.c:304 sysv/memory: did not match transport list
[g090:1261044] common_ucx.c:304 self/memory0: did not match transport list
[g090:1261044] common_ucx.c:304 tcp/lo: did not match transport list
[g090:1261044] common_ucx.c:304 tcp/eth0: did not match transport list
[g090:1261044] common_ucx.c:304 tcp/ib0: did not match transport list
[g091:1549666] common_ucx.c:304 posix/memory: did not match transport list
[g091:1549666] common_ucx.c:304 sysv/memory: did not match transport list
[g091:1549666] common_ucx.c:304 self/memory0: did not match transport list
[g091:1549666] common_ucx.c:304 tcp/ib0: did not match transport list
[g091:1549666] common_ucx.c:304 tcp/lo: did not match transport list
[g091:1549666] common_ucx.c:304 tcp/eth0: did not match transport list
[g090:1261043] common_ucx.c:304 tcp/ib0: did not match transport list
[g091:1549665] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g091:1549665] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g091:1549665] common_ucx.c:311 support level is transports and devices
[g091:1549665] pml_ucx.c:289 mca_pml_ucx_init
[g090:1261044] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g090:1261044] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g090:1261044] common_ucx.c:311 support level is transports and devices
[g090:1261044] pml_ucx.c:289 mca_pml_ucx_init
[g090:1261043] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g090:1261043] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g090:1261043] common_ucx.c:311 support level is transports and devices
[g090:1261043] pml_ucx.c:289 mca_pml_ucx_init
[g091:1549666] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g091:1549666] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g091:1549666] common_ucx.c:311 support level is transports and devices
[g091:1549666] pml_ucx.c:289 mca_pml_ucx_init
[g093:1949867] common_ucx.c:304 posix/memory: did not match transport list
[g093:1949867] common_ucx.c:304 sysv/memory: did not match transport list
[g093:1949867] common_ucx.c:304 self/memory0: did not match transport list
[g093:1949867] common_ucx.c:304 tcp/ib0: did not match transport list
[g093:1949867] common_ucx.c:304 tcp/lo: did not match transport list
[g093:1949867] common_ucx.c:304 tcp/eth0: did not match transport list
[g093:1949867] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g093:1949867] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g093:1949867] common_ucx.c:311 support level is transports and devices
[g093:1949867] pml_ucx.c:289 mca_pml_ucx_init
[g092:1207223] common_ucx.c:304 posix/memory: did not match transport list
[g092:1207223] common_ucx.c:304 sysv/memory: did not match transport list
[g092:1207223] common_ucx.c:304 self/memory0: did not match transport list
[g092:1207223] common_ucx.c:304 tcp/eth0: did not match transport list
[g092:1207223] common_ucx.c:304 tcp/lo: did not match transport list
[g092:1207223] common_ucx.c:304 tcp/ib0: did not match transport list
[g092:1207223] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g092:1207223] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g092:1207223] common_ucx.c:311 support level is transports and devices
[g092:1207223] pml_ucx.c:289 mca_pml_ucx_init
[g091:1549665] pml_ucx.c:114 Pack remote worker address, size 238
[g091:1549665] pml_ucx.c:114 Pack local worker address, size 374
[g091:1549665] pml_ucx.c:351 created ucp context 0x76adf0, worker 0x7d9f80
[g091:1549665] pml_ucx_component.c:129 returning priority 51
[g091:1549666] pml_ucx.c:114 Pack remote worker address, size 238
[g091:1549666] pml_ucx.c:114 Pack local worker address, size 374
[g091:1549666] pml_ucx.c:351 created ucp context 0x769d30, worker 0x7d99e0
[g091:1549666] pml_ucx_component.c:129 returning priority 51
[g090:1261043] pml_ucx.c:114 Pack remote worker address, size 238
[g090:1261044] pml_ucx.c:114 Pack remote worker address, size 238
[g090:1261043] pml_ucx.c:114 Pack local worker address, size 374
[g090:1261043] pml_ucx.c:351 created ucp context 0x76afd0, worker 0x7da140
[g090:1261043] pml_ucx_component.c:129 returning priority 51
[g090:1261044] pml_ucx.c:114 Pack local worker address, size 374
[g090:1261044] pml_ucx.c:351 created ucp context 0x769eb0, worker 0x7d97d0
[g090:1261044] pml_ucx_component.c:129 returning priority 51
[g093:1949867] pml_ucx.c:114 Pack remote worker address, size 238
[g093:1949867] pml_ucx.c:114 Pack local worker address, size 374
[g093:1949867] pml_ucx.c:351 created ucp context 0x768520, worker 0x7d7dc0
[g093:1949867] pml_ucx_component.c:129 returning priority 51
[g092:1207223] pml_ucx.c:114 Pack remote worker address, size 238
[g092:1207223] pml_ucx.c:114 Pack local worker address, size 374
[g092:1207223] pml_ucx.c:351 created ucp context 0x768520, worker 0x7d7f20
[g092:1207223] pml_ucx_component.c:129 returning priority 51
[g090:1261043] pml_ucx.c:182 Got proc 0 address, size 374
[g090:1261043] pml_ucx.c:411 connecting to proc. 0
[g090:1261044] pml_ucx.c:182 Got proc 1 address, size 374
[g090:1261044] pml_ucx.c:411 connecting to proc. 1
[g091:1549666] pml_ucx.c:182 Got proc 3 address, size 374
[g091:1549666] pml_ucx.c:411 connecting to proc. 3
[g092:1207223] pml_ucx.c:182 Got proc 4 address, size 374
[g092:1207223] pml_ucx.c:411 connecting to proc. 4
[g093:1949867] pml_ucx.c:182 Got proc 5 address, size 374
[g093:1949867] pml_ucx.c:411 connecting to proc. 5
[g091:1549665] pml_ucx.c:182 Got proc 2 address, size 374
[g091:1549665] pml_ucx.c:411 connecting to proc. 2
[g090:1261043] pml_ucx.c:182 Got proc 1 address, size 374
[g090:1261043] pml_ucx.c:411 connecting to proc. 1
[g090:1261044] pml_ucx.c:182 Got proc 0 address, size 374
[g090:1261044] pml_ucx.c:411 connecting to proc. 0
[g091:1549666] pml_ucx.c:182 Got proc 2 address, size 374
[g091:1549666] pml_ucx.c:411 connecting to proc. 2
[g091:1549665] pml_ucx.c:182 Got proc 3 address, size 374
[g091:1549665] pml_ucx.c:411 connecting to proc. 3
-------------------------------------------------------------
STREAM version $Revision: 1.8 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Total Aggregate Array size = 105000000 (elements)
Total Aggregate Memory per array = 801.1 MiB (= 0.8 GiB).
Total Aggregate memory required = 2403.3 MiB (= 2.3 GiB).
Data is distributed across 6 MPI ranks
   Array size per MPI rank = 17500000 (elements)
   Memory per array per MPI rank = 133.5 MiB (= 0.1 GiB).
   Total memory per MPI rank = 400.5 MiB (= 0.4 GiB).
-------------------------------------------------------------
Each kernel will be executed 10 times.
 The *best* time for each kernel (excluding the first iteration)
 will be used to compute the reported bandwidth.
The SCALAR value used for this run is 0.420000
-------------------------------------------------------------
Your timer granularity/precision appears to be 1 microseconds.
[g091:1549666] pml_ucx.c:182 Got proc 0 address, size 238
[g091:1549666] pml_ucx.c:411 connecting to proc. 0
[g093:1949867] pml_ucx.c:182 Got proc 0 address, size 238
[g093:1949867] pml_ucx.c:411 connecting to proc. 0
[g092:1207223] pml_ucx.c:182 Got proc 0 address, size 238
[g092:1207223] pml_ucx.c:411 connecting to proc. 0
[g091:1549665] pml_ucx.c:182 Got proc 0 address, size 238
[g091:1549665] pml_ucx.c:411 connecting to proc. 0
Each test below will take on the order of 14543 microseconds.
   (= 14543 timer ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 timer ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
[g090:1261043] pml_ucx.c:182 Got proc 2 address, size 238
[g090:1261043] pml_ucx.c:411 connecting to proc. 2
[g090:1261043] pml_ucx.c:182 Got proc 3 address, size 238
[g090:1261043] pml_ucx.c:411 connecting to proc. 3
[g090:1261043] pml_ucx.c:182 Got proc 4 address, size 238
[g090:1261043] pml_ucx.c:411 connecting to proc. 4
[g090:1261043] pml_ucx.c:182 Got proc 5 address, size 238
[g090:1261043] pml_ucx.c:411 connecting to proc. 5
[g093:1949867] pml_ucx.c:182 Got proc 4 address, size 238
[g093:1949867] pml_ucx.c:411 connecting to proc. 4
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:          83323.4     0.029720     0.020162     0.048685
Scale:         81398.2     0.028492     0.020639     0.043052
Add:           86355.2     0.040145     0.029182     0.049721
Triad:         86804.4     0.039043     0.029031     0.047365
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------
[g090:1261043] common_ucx.c:440 disconnecting from rank 0
[g090:1261044] common_ucx.c:440 disconnecting from rank 1
[g093:1949867] common_ucx.c:440 disconnecting from rank 5
[g092:1207223] common_ucx.c:440 disconnecting from rank 4
[g091:1549666] common_ucx.c:440 disconnecting from rank 2
[g091:1549665] common_ucx.c:440 disconnecting from rank 0
[g090:1261043] common_ucx.c:440 disconnecting from rank 1
[g090:1261043] common_ucx.c:440 disconnecting from rank 2
[g090:1261044] common_ucx.c:440 disconnecting from rank 0
[g090:1261044] common_ucx.c:404 waiting for 0 disconnect requests
[g090:1261043] common_ucx.c:404 waiting for 1 disconnect requests
[g090:1261043] common_ucx.c:440 disconnecting from rank 3
[g090:1261043] common_ucx.c:404 waiting for 1 disconnect requests
[g090:1261043] common_ucx.c:440 disconnecting from rank 4
[g090:1261043] common_ucx.c:404 waiting for 1 disconnect requests
[g090:1261043] common_ucx.c:440 disconnecting from rank 5
[g090:1261043] common_ucx.c:404 waiting for 1 disconnect requests
[g090:1261043] common_ucx.c:404 waiting for 0 disconnect requests
[g091:1549665] common_ucx.c:404 waiting for 1 disconnect requests
[g092:1207223] common_ucx.c:440 disconnecting from rank 0
[g092:1207223] common_ucx.c:404 waiting for 1 disconnect requests
[g092:1207223] common_ucx.c:404 waiting for 0 disconnect requests
[g091:1549665] common_ucx.c:440 disconnecting from rank 2
[g091:1549665] common_ucx.c:440 disconnecting from rank 3
[g093:1949867] common_ucx.c:440 disconnecting from rank 0
[g093:1949867] common_ucx.c:404 waiting for 1 disconnect requests
[g093:1949867] common_ucx.c:440 disconnecting from rank 4
[g093:1949867] common_ucx.c:404 waiting for 1 disconnect requests
[g093:1949867] common_ucx.c:404 waiting for 0 disconnect requests
[g091:1549665] common_ucx.c:404 waiting for 0 disconnect requests
[g091:1549666] common_ucx.c:440 disconnecting from rank 3
[g091:1549666] common_ucx.c:440 disconnecting from rank 0
[g091:1549666] common_ucx.c:404 waiting for 1 disconnect requests
[g091:1549666] common_ucx.c:404 waiting for 0 disconnect requests
[g090:1261044] pml_ucx.c:367 mca_pml_ucx_cleanup
[g090:1261043] pml_ucx.c:367 mca_pml_ucx_cleanup
[g091:1549665] pml_ucx.c:367 mca_pml_ucx_cleanup
[g093:1949867] pml_ucx.c:367 mca_pml_ucx_cleanup
[g091:1549666] pml_ucx.c:367 mca_pml_ucx_cleanup
[g092:1207223] pml_ucx.c:367 mca_pml_ucx_cleanup
[g090:1261044] pml_ucx.c:268 mca_pml_ucx_close
[g091:1549665] pml_ucx.c:268 mca_pml_ucx_close
[g091:1549666] pml_ucx.c:268 mca_pml_ucx_close
[g090:1261043] pml_ucx.c:268 mca_pml_ucx_close
[g092:1207223] pml_ucx.c:268 mca_pml_ucx_close
[g093:1949867] pml_ucx.c:268 mca_pml_ucx_close

@yosefe
Copy link
Contributor

yosefe commented Sep 25, 2022

@RobbieTheK i don't see the error "PML ucx cannot be selected" here, seems UCX is used.
also, could you pls run "ucx_info -v" and post the output?

@LinuxPersonEC
Copy link
Author

We don't have ucx installed, isn't it available in OpenMPI 4.1.1? From the verbose output does this apply? UCX version 1.11.0

ucx_info -v
-bash: ucx_info: command not found

Here's what I get with OSU's benchmark:

mpirun  -mca pml_ucx_verbose 999 --mca opal_warn_on_missing_libcuda 0  --mca orte_base_help_aggregate 0 --host xxx,xxx --oversubscribe -n 2 ./osu_latency_mp 
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   xxx
  Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   xxx
  Local device: mlx5_0
--------------------------------------------------------------------------
[g171:2707658] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g171:2707659] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g171:2707658] common_ucx.c:304 posix/memory: did not match transport list
[g171:2707658] common_ucx.c:304 sysv/memory: did not match transport list
[g171:2707658] common_ucx.c:304 self/memory0: did not match transport list
[g171:2707658] common_ucx.c:304 tcp/lo: did not match transport list
[g171:2707658] common_ucx.c:304 tcp/eth0: did not match transport list
[g171:2707658] common_ucx.c:304 tcp/ib0: did not match transport list
[g171:2707658] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g171:2707658] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g171:2707658] common_ucx.c:311 support level is transports and devices
[g171:2707658] pml_ucx.c:289 mca_pml_ucx_init
[g171:2707659] common_ucx.c:304 posix/memory: did not match transport list
[g171:2707659] common_ucx.c:304 sysv/memory: did not match transport list
[g171:2707659] common_ucx.c:304 self/memory0: did not match transport list
[g171:2707659] common_ucx.c:304 tcp/lo: did not match transport list
[g171:2707659] common_ucx.c:304 tcp/eth0: did not match transport list
[g171:2707659] common_ucx.c:304 tcp/ib0: did not match transport list
[g171:2707659] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g171:2707659] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g171:2707659] common_ucx.c:311 support level is transports and devices
[g171:2707659] pml_ucx.c:289 mca_pml_ucx_init
[g171:2707658] pml_ucx.c:114 Pack remote worker address, size 238
[g171:2707658] pml_ucx.c:114 Pack local worker address, size 374
[g171:2707658] pml_ucx.c:351 created ucp context 0x78dcd0, worker 0x7fce80
[g171:2707658] pml_ucx_component.c:129 returning priority 51
[g171:2707659] pml_ucx.c:114 Pack remote worker address, size 238
[g171:2707659] pml_ucx.c:114 Pack local worker address, size 374
[g171:2707659] pml_ucx.c:351 created ucp context 0x78c950, worker 0x7fc0b0
[g171:2707659] pml_ucx_component.c:129 returning priority 51
[g171:2707658] pml_ucx.c:182 Got proc 0 address, size 374
[g171:2707658] pml_ucx.c:411 connecting to proc. 0
[g171:2707659] pml_ucx.c:182 Got proc 1 address, size 374
[g171:2707659] pml_ucx.c:411 connecting to proc. 1
[g171:2707658] pml_ucx.c:182 Got proc 1 address, size 374
[g171:2707658] pml_ucx.c:411 connecting to proc. 1
[g171:2707659] pml_ucx.c:182 Got proc 0 address, size 374
[g171:2707659] pml_ucx.c:411 connecting to proc. 0
# Number of forked processes in sender: 2
# Number of forked processes in receiver: 2
# OSU MPI Multi-process Latency Test v6.1
# Size          Latency (us)
0               0.27
1               0.27
2               0.27
4               0.27
8               0.27
16              0.27
32              0.28
64              0.30
128             0.36
256             0.39
512             0.45
1024            0.56
2048            0.74
4096            1.09
8192            1.78
16384           2.89
32768           4.61
65536           8.07
131072         14.87
262144         11.70
524288         21.96
1048576        46.74
2097152       133.04
4194304       293.98
[g171:2707659] common_ucx.c:440 disconnecting from rank 1
[g171:2707658] common_ucx.c:440 disconnecting from rank 0
[g171:2707658] common_ucx.c:440 disconnecting from rank 1
[g171:2707658] common_ucx.c:404 waiting for 0 disconnect requests
[g171:2707659] common_ucx.c:440 disconnecting from rank 0
[g171:2707659] common_ucx.c:404 waiting for 0 disconnect requests
[g171:2707658] pml_ucx.c:367 mca_pml_ucx_cleanup
[g171:2707659] pml_ucx.c:367 mca_pml_ucx_cleanup
[g171:2707659] pml_ucx.c:268 mca_pml_ucx_close
[g171:2707658] pml_ucx.c:268 mca_pml_ucx_close

@yosefe
Copy link
Contributor

yosefe commented Sep 25, 2022

OpenMPI was probably build with custom UCX path, need to run ucx_info from there
Anyhow, according to this output UCX is being used correctly

@LinuxPersonEC
Copy link
Author

OK so my questions are why the WARNING: There was an error initializing an OpenFabrics device and segfault when using --mca btl openib and does Device vendor ID: 0x02c9 need to be added to the .ini file?

How would I find ucx here?

openmpi-4.1.1_ucx_cuda_11.0.3_support]$ find . -name *ucx* 
./lib/openmpi/mca_sshmem_ucx.so
./lib/openmpi/mca_spml_ucx.so
./lib/openmpi/mca_osc_ucx.la
./lib/openmpi/mca_sshmem_ucx.la
./lib/openmpi/mca_osc_ucx.so
./lib/openmpi/mca_pml_ucx.la
./lib/openmpi/mca_atomic_ucx.so
./lib/openmpi/mca_atomic_ucx.la
./lib/openmpi/mca_spml_ucx.la
./lib/openmpi/mca_pml_ucx.so
./lib/libmca_common_ucx.la
./lib/libmca_common_ucx.so.40
./lib/libmca_common_ucx.so.40.30.1
./lib/libmca_common_ucx.so

@yosefe
Copy link
Contributor

yosefe commented Sep 25, 2022

OK so my questions are why the WARNING: There was an error initializing an OpenFabrics device and segfault when using --mca btl openib and does Device vendor ID: 0x02c9 need to be added to the .ini file?

I'd advise to disable btl/openib component since it was deprecated and replaced by ucx: add -mca btl ^openib to mpirun.

How would I find ucx here?

ldd ./lib/openmpi/mca_pml_ucx.so|grep libucp

@LinuxPersonEC
Copy link
Author

LinuxPersonEC commented Sep 25, 2022 •
edited
Loading

I'd advise to disable btl/openib component since it was deprecated and replaced by ucx: add -mca btl ^openib to mpirun

Thanks was aware of that just was suggesting a more useful message/log rather than segfault.

Does anyone know of a way to better use these benchmarks to find if there is a performance difference, e.g., if one's faster than the other? I get very similar results using STREAMS and OSU's micro benchmarks.

also:

ldd ./lib/openmpi/mca_pml_ucx.so|grep libucp
	libucp.so.0 => /burg/opt/ucx_cuda_11.0.3_support/lib/libucp.so.0 (0x0000155554440000)

But is there a way to run ucx standalone when it's compiled in OpenMPI?

@LinuxPersonEC
Copy link
Author

@RobbieTheK i don't see the error "PML ucx cannot be selected" here, seems UCX is used.

This error happens when I use the OSU Multi-threaded Latency Test and likely, OpenMPI was not compiled with OCX support. Can I make a suggestion to improve the warning to mention that?

@janjust
Copy link
Contributor

janjust commented Oct 11, 2022 via email

@bbsy789 bbsy789 mentioned this issue Oct 28, 2022
Closed
@wangyugui-e16
Copy link

another one.

WARNING: No preset parameters were found for the device that Open MPI detected:
Device name: irdma0
Device vendor ID: 0x8086
Device vendor part ID: 5522

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Target: v4.1.x
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jsquyres @yosefe @janjust @LinuxPersonEC @wangyugui-e16