ompi
ompi copied to clipboard
open-mpi
Device vendor ID: 0x02c9, part ID: 4123, WARNING: No preset parameters were found for the device that Open MPI detected
OpenMPI 4.1.1 in RHEL 8, 5e:00.0 Infiniband controller [0207]: Mellanox Technologies MT28908 Family [ConnectX-6] [15b3:101b]
Using this STREAM benchmark here are some verbose logs:
mpirun -mca pml_base_verbose 100 -mca btl_base_verbose 100 -mca mca_base_verbose 100 -mca btl_openib_verbose true -mca pml ucx --mca orte_base_help_aggregate 0 --mca opal_warn_on_missing_libcuda 0 -np 1 --oversubscribe ./stream_mpi --oversubscribe
[g183:270526] mca: base: components_register: registering framework btl components
[g183:270526] mca: base: components_register: found loaded component openib
[g183:270526] mca: base: components_register: component openib register function successful
[g183:270526] mca: base: components_register: found loaded component sm
[g183:270526] mca: base: components_register: found loaded component tcp
[g183:270526] mca: base: components_register: component tcp register function successful
[g183:270526] mca: base: components_register: found loaded component self
[g183:270526] mca: base: components_register: component self register function successful
[g183:270526] mca: base: components_register: found loaded component vader
[g183:270526] mca: base: components_register: component vader register function successful
[g183:270526] mca: base: components_register: found loaded component smcuda
[g183:270526] mca: base: components_register: component smcuda register function successful
[g183:270526] mca: base: components_open: opening btl components
[g183:270526] mca: base: components_open: found loaded component openib
[g183:270526] mca: base: components_open: component openib open function successful
[g183:270526] mca: base: components_open: found loaded component tcp
[g183:270526] mca: base: components_open: component tcp open function successful
[g183:270526] mca: base: components_open: found loaded component self
[g183:270526] mca: base: components_open: component self open function successful
[g183:270526] mca: base: components_open: found loaded component vader
[g183:270526] mca: base: components_open: component vader open function successful
[g183:270526] mca: base: components_open: found loaded component smcuda
[g183:270526] btl: smcuda: cuda_max_send_size=131072, max_send_size=32768, max_frag_size=131072
[g183:270526] mca: base: components_open: component smcuda open function successful
[g183:270526] select: initializing btl component openib
[g183:270526] Checking distance from this process to device=mlx5_0
[g183:270526] hwloc_distances->nbobjs=4
[g183:270526] hwloc_distances->values[0]=10
[g183:270526] hwloc_distances->values[1]=21
[g183:270526] hwloc_distances->values[2]=11
[g183:270526] hwloc_distances->values[3]=21
[g183:270526] ibv_obj->type set to NULL
[g183:270526] Process is bound: distance to device is 0.000000
[g183][[11854,1],0][btl_openib_ini.c:172:opal_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 4123
[g183][[11854,1],0][btl_openib_ini.c:188:opal_btl_openib_ini_query] Found corresponding INI values: Mellanox ConnectX6
[g183][[11854,1],0][btl_openib_ini.c:172:opal_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0
[g183][[11854,1],0][btl_openib_ini.c:188:opal_btl_openib_ini_query] Found corresponding INI values: default
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: xxxx
Local device: mlx5_0
--------------------------------------------------------------------------
[g183:270526] select: init of component openib returned failure
[g183:270526] mca: base: close: component openib closed
[g183:270526] mca: base: close: unloading component openib
[g183:270526] select: initializing btl component tcp
[g183:270526] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[g183:270526] btl: tcp: Found match: 127.0.0.1 (lo)
[g183:270526] btl:tcp: Attempting to bind to AF_INET port 1024
[g183:270526] btl:tcp: Successfully bound to AF_INET port 1024
[g183:270526] btl:tcp: my listening v4 socket is 0.0.0.0:1024
[g183:270526] btl:tcp: examining interface eth0
[g183:270526] btl:tcp: using ipv6 interface eth0
[g183:270526] btl:tcp: examining interface ib0
[g183:270526] btl:tcp: using ipv6 interface ib0
[g183:270526] select: init of component tcp returned success
[g183:270526] select: initializing btl component self
[g183:270526] select: init of component self returned success
[g183:270526] select: initializing btl component vader
[g183:270526] select: init of component vader returned failure
[g183:270526] mca: base: close: component vader closed
[g183:270526] mca: base: close: unloading component vader
[g183:270526] select: initializing btl component smcuda
[g183:270526] select: init of component smcuda returned failure
[g183:270526] mca: base: close: component smcuda closed
[g183:270526] mca: base: close: unloading component smcuda
[g183:270526] mca: base: components_register: registering framework pml components
[g183:270526] mca: base: components_register: found loaded component ucx
[g183:270526] mca: base: components_register: component ucx register function successful
[g183:270526] mca: base: components_open: opening pml components
[g183:270526] mca: base: components_open: found loaded component ucx
[g183:270526] mca: base: components_open: component ucx open function successful
[g183:270526] select: initializing pml component ucx
[g183:270526] select: init returned priority 51
[g183:270526] selected ucx best priority 51
[g183:270526] select: component ucx selected
-------------------------------------------------------------
STREAM version $Revision: 1.8 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Total Aggregate Array size = 105000000 (elements)
Total Aggregate Memory per array = 801.1 MiB (= 0.8 GiB).
Total Aggregate memory required = 2403.3 MiB (= 2.3 GiB).
Data is distributed across 1 MPI ranks
Array size per MPI rank = 105000000 (elements)
Memory per array per MPI rank = 801.1 MiB (= 0.8 GiB).
Total memory per MPI rank = 2403.3 MiB (= 2.3 GiB).
-------------------------------------------------------------
Each kernel will be executed 10 times.
The *best* time for each kernel (excluding the first iteration)
will be used to compute the reported bandwidth.
The SCALAR value used for this run is 0.420000
-------------------------------------------------------------
Your timer granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 68148 microseconds.
(= 68148 timer ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 timer ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function Best Rate MB/s Avg time Min time Max time
Copy: 13648.7 0.123378 0.123089 0.123952
Scale: 13784.7 0.122519 0.121874 0.123266
Add: 14363.8 0.175696 0.175440 0.175882
Triad: 14216.1 0.177668 0.177264 0.178539
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------
[g183:270526] mca: base: close: component ucx closed
[g183:270526] mca: base: close: unloading component ucx
[g183:270526] mca: base: close: component tcp closed
[g183:270526] mca: base: close: unloading component tcp
[g183:270526] mca: base: close: component self closed
[g183:270526] mca: base: close: unloading component self
WARNING: No preset parameters were found for the device that Open MPI detected:
Local host: xxxx
Device name: mlx5_0
Device vendor ID: 0x02c9
Device vendor part ID: 4123
I found a reference to this in the comments for mca-btl-openib-device-params.ini
# Note: Several vendors resell Mellanox hardware and put their own firmware
# on the cards, therefore overriding the default Mellanox vendor ID.
#
# Mellanox 0x02c9
Forcing ucx still generates the warning:
mpirun -mca pml ucx --mca orte_base_help_aggregate 0 --mca opal_warn_on_missing_libcuda 0 -np 1 --oversubscribe ./stream_mpi --oversubscribe
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: xxxx
Local device: mlx5_0
--------------------------------------------------------------------------
-------------------------------------------------------------
STREAM version $Revision: 1.8 $
With version 4.1.1, if I use --mca btl 'openib' I get seg faults which I believe is expected as it's deprecated. I've tried --mca btl '^openib', --mca btl 'tcp' (or --mca btl 'tcp,self' using the OSU BMs) and the benchmark results are very similar even when I use multiple CPUs, threads and/or nodes. They also run without the warning messages. If I don't use a --mca option, I get the WARNING: message.
Does anyone know of a tried and true way to run these benchmarks so know if these MCA parameters make a difference or am I just not understanding how to use these? Perhaps running these benchmarks on a very active cluster with shared CPUs/nodes will affect the results?
UCX is definitely the way to go; this is NVIDIA's preferred mechanism.
Sidenote: running with --mca btl openib is expected to fail, because you also need the self and vader BTLs for self-process-loopback and on-node-process communication, respectively. That being said, it seems a bit anti-social to segv instead of printing a helpful error message in these cases. ☹️
The warning message is definitely coming from the openib BTL (I just double checked that by checking for that message in the v4.1.x source -- it's only in the openib BTL).
I think you should be able to run with:
$ mpirun --mca pml ucx --mca btl '^openib' ...
This tells Open MPI to use the UCX PML for point-to-point MPI communication, and to skip using the openib BTL in any other contexts that it might come up (E.g., one sided MPI communication).
Any further suggestions from the @open-mpi/ucx team?
That being said, it seems a bit anti-social to segv instead of printing a helpful error message in these cases. ☹️
Here's what it looks like running mpirun --mca btl 'openib' -np 4 --host xxx,yyy --oversubscribe --mca orte_base_help_aggregate 0 --mca opal_warn_on_missing_libcuda 0 ./stream_mpi
ucx_cuda_11.0.3_support/lib/libucs.so.0(+0x2b138) [0x155541ff9138]
=================================
/lib64/libpthread.so.0(+0x12dd0)[0x155554df7dd0]
[g123:119486] *** End of error message ***
[g123:119485] *** Process received signal ***
[g123:119485] Signal: Segmentation fault (11)
[g123:119485] Signal code: (-6)
[g123:119485] Failing at address: 0x859d90001d2bd
==== backtrace (tid: 119487) ====
0 /burg/opt/ucx_cuda_11.0.3_support/lib/libucs.so.0(ucs_handle_error+0x254) [0x155541ff8cd4]
1 /burg/opt/ucx_cuda_11.0.3_support/lib/libucs.so.0(+0x2ae8c) [0x155541ff8e8c]
2 /burg/opt/ucx_cuda_11.0.3_support/lib/libucs.so.0(+0x2b138) [0x155541ff9138]
=================================
[g123:119485] [ 0] /lib64/libpthread.so.0(+0x12dd0)[0x155554df7dd0]
[g123:119485] *** End of error message ***
[g123:119487] *** Process received signal ***
[g123:119487] Signal: Segmentation fault (11)
[g123:119487] Signal code: (-6)
[g123:119487] Failing at address: 0x859d90001d2bf
[g123:119487] [ 0] /lib64/libpthread.so.0(+0x12dd0)[0x155554df7dd0]
[g123:119487] *** End of error message ***
--------------------------------------------------------------------------
And perhaps of interest the warning is clearer and no segv when running:
mpirun --mca btl 'openib' --mca pml ucx -np 4 --host g123,g124 --oversubscribe --mca orte_base_help_aggregate 0 --mca opal_warn_on_missing_libcuda 0 ./stream_mpi
WARNING: There was an error initializing an OpenFabrics device.
Local host: xxxx
Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
No components were able to be opened in the btl framework.
This typically means that either no components of this type were installed, or none of the installed components can be loaded. Sometimes this means that shared libraries required by these components are unable to be found/loaded.
Is there an option/parameter to verify ucx is available?
Also when running mpirun --mca pml ucx --mca btl '^openib' on the OSU Multi-threaded Latency Test why can't ucx be selected?
[xxx:105832] PML ucx cannot be selected
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.
This typically means that either no components of this type were installed, or none of the installed components can be loaded. Sometimes this means that shared libraries required by these components are unable to be found/loaded.
Host: xxx
Framework: pml
--------------------------------------------------------------------------
[xxx:2328334] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2198
UCX built with multithreaded support?
— Tomislav Janjusic
From: RobbieTheK @.> Sent: Friday, September 23, 2022 3:22:05 PM To: open-mpi/ompi @.> Cc: Tomislav Janjusic @.>; Team mention @.> Subject: Re: [open-mpi/ompi] Device vendor ID: 0x02c9, part ID: 4123, WARNING: No preset parameters were found for the device that Open MPI detected (Issue #10841)
Also when running mpirun --mca pml ucx --mca btl '^openib' on the OSU Multi-threaded Latency Testhttps://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmvapich.cse.ohio-state.edu%2Fbenchmarks%2F&data=05%7C01%7Ctomislavj%40nvidia.com%7C020d2cfff7bc459c16f508da9da148cd%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637995613283417872%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8d85VqcBgwu262lMPqJSJr7Uxh785OfFt%2FydWbNo0NU%3D&reserved=0 why can't ucx be selected?
[xxx:105832] PML ucx cannot be selected
No components were able to be opened in the pml framework.
This typically means that either no components of this type were installed, or none of the installed components can be loaded. Sometimes this means that shared libraries required by these components are unable to be found/loaded.
Host: xxx Framework: pml
[xxx:2328334] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2198
— Reply to this email directly, view it on GitHubhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-mpi%2Fompi%2Fissues%2F10841%23issuecomment-1256637707&data=05%7C01%7Ctomislavj%40nvidia.com%7C020d2cfff7bc459c16f508da9da148cd%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637995613283417872%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SQV1ct6jpYva9orgezQYKVvikHDXNPms%2Bp3ZQYKTzig%3D&reserved=0, or unsubscribehttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAZCSJDZFCNFKG7I4JJKV7DV7YGO3ANCNFSM6AAAAAAQUBJVX4&data=05%7C01%7Ctomislavj%40nvidia.com%7C020d2cfff7bc459c16f508da9da148cd%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637995613283417872%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=c1exQtjFSQsIWEij55gdlpGLEiLpXAjxSqkvpnY%2BtkI%3D&reserved=0. You are receiving this because you are on a team that was mentioned.Message ID: @.***>
UCX built with multithreaded support?
Ah good question was done before my time. Is there a way to use -x UCX_LOG_LEVEL=data to find out?
I have a dim recollection that there were some bugs with UCX/UCT in the past that caused segvs when used in specific combinations. Do you have the most recent version of UCX?
@RobbieTheK can you pls run openmpi with "-mca pml_ucx_verbose 999" to show why UCX is not selected?
mpirun -mca pml_ucx_verbose 999 -np 6 --host xxx,xxx,xxx,xxx --oversubscribe --mca orte_base_help_aggregate 0 --mca opal_warn_on_missing_libcuda 0 ./stream_mpi
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: xxx
Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: xx
Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: xxx
Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: xxx
Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: xxxx
Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: xxx
Local device: mlx5_0
--------------------------------------------------------------------------
[g090:1261043] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g090:1261044] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g091:1549665] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g091:1549666] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g093:1949867] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g092:1207223] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g091:1549665] common_ucx.c:304 posix/memory: did not match transport list
[g091:1549665] common_ucx.c:304 sysv/memory: did not match transport list
[g091:1549665] common_ucx.c:304 self/memory0: did not match transport list
[g091:1549665] common_ucx.c:304 tcp/ib0: did not match transport list
[g091:1549665] common_ucx.c:304 tcp/lo: did not match transport list
[g091:1549665] common_ucx.c:304 tcp/eth0: did not match transport list
[g090:1261043] common_ucx.c:304 posix/memory: did not match transport list
[g090:1261043] common_ucx.c:304 sysv/memory: did not match transport list
[g090:1261043] common_ucx.c:304 self/memory0: did not match transport list
[g090:1261043] common_ucx.c:304 tcp/lo: did not match transport list
[g090:1261043] common_ucx.c:304 tcp/eth0: did not match transport list
[g090:1261044] common_ucx.c:304 posix/memory: did not match transport list
[g090:1261044] common_ucx.c:304 sysv/memory: did not match transport list
[g090:1261044] common_ucx.c:304 self/memory0: did not match transport list
[g090:1261044] common_ucx.c:304 tcp/lo: did not match transport list
[g090:1261044] common_ucx.c:304 tcp/eth0: did not match transport list
[g090:1261044] common_ucx.c:304 tcp/ib0: did not match transport list
[g091:1549666] common_ucx.c:304 posix/memory: did not match transport list
[g091:1549666] common_ucx.c:304 sysv/memory: did not match transport list
[g091:1549666] common_ucx.c:304 self/memory0: did not match transport list
[g091:1549666] common_ucx.c:304 tcp/ib0: did not match transport list
[g091:1549666] common_ucx.c:304 tcp/lo: did not match transport list
[g091:1549666] common_ucx.c:304 tcp/eth0: did not match transport list
[g090:1261043] common_ucx.c:304 tcp/ib0: did not match transport list
[g091:1549665] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g091:1549665] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g091:1549665] common_ucx.c:311 support level is transports and devices
[g091:1549665] pml_ucx.c:289 mca_pml_ucx_init
[g090:1261044] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g090:1261044] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g090:1261044] common_ucx.c:311 support level is transports and devices
[g090:1261044] pml_ucx.c:289 mca_pml_ucx_init
[g090:1261043] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g090:1261043] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g090:1261043] common_ucx.c:311 support level is transports and devices
[g090:1261043] pml_ucx.c:289 mca_pml_ucx_init
[g091:1549666] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g091:1549666] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g091:1549666] common_ucx.c:311 support level is transports and devices
[g091:1549666] pml_ucx.c:289 mca_pml_ucx_init
[g093:1949867] common_ucx.c:304 posix/memory: did not match transport list
[g093:1949867] common_ucx.c:304 sysv/memory: did not match transport list
[g093:1949867] common_ucx.c:304 self/memory0: did not match transport list
[g093:1949867] common_ucx.c:304 tcp/ib0: did not match transport list
[g093:1949867] common_ucx.c:304 tcp/lo: did not match transport list
[g093:1949867] common_ucx.c:304 tcp/eth0: did not match transport list
[g093:1949867] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g093:1949867] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g093:1949867] common_ucx.c:311 support level is transports and devices
[g093:1949867] pml_ucx.c:289 mca_pml_ucx_init
[g092:1207223] common_ucx.c:304 posix/memory: did not match transport list
[g092:1207223] common_ucx.c:304 sysv/memory: did not match transport list
[g092:1207223] common_ucx.c:304 self/memory0: did not match transport list
[g092:1207223] common_ucx.c:304 tcp/eth0: did not match transport list
[g092:1207223] common_ucx.c:304 tcp/lo: did not match transport list
[g092:1207223] common_ucx.c:304 tcp/ib0: did not match transport list
[g092:1207223] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g092:1207223] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g092:1207223] common_ucx.c:311 support level is transports and devices
[g092:1207223] pml_ucx.c:289 mca_pml_ucx_init
[g091:1549665] pml_ucx.c:114 Pack remote worker address, size 238
[g091:1549665] pml_ucx.c:114 Pack local worker address, size 374
[g091:1549665] pml_ucx.c:351 created ucp context 0x76adf0, worker 0x7d9f80
[g091:1549665] pml_ucx_component.c:129 returning priority 51
[g091:1549666] pml_ucx.c:114 Pack remote worker address, size 238
[g091:1549666] pml_ucx.c:114 Pack local worker address, size 374
[g091:1549666] pml_ucx.c:351 created ucp context 0x769d30, worker 0x7d99e0
[g091:1549666] pml_ucx_component.c:129 returning priority 51
[g090:1261043] pml_ucx.c:114 Pack remote worker address, size 238
[g090:1261044] pml_ucx.c:114 Pack remote worker address, size 238
[g090:1261043] pml_ucx.c:114 Pack local worker address, size 374
[g090:1261043] pml_ucx.c:351 created ucp context 0x76afd0, worker 0x7da140
[g090:1261043] pml_ucx_component.c:129 returning priority 51
[g090:1261044] pml_ucx.c:114 Pack local worker address, size 374
[g090:1261044] pml_ucx.c:351 created ucp context 0x769eb0, worker 0x7d97d0
[g090:1261044] pml_ucx_component.c:129 returning priority 51
[g093:1949867] pml_ucx.c:114 Pack remote worker address, size 238
[g093:1949867] pml_ucx.c:114 Pack local worker address, size 374
[g093:1949867] pml_ucx.c:351 created ucp context 0x768520, worker 0x7d7dc0
[g093:1949867] pml_ucx_component.c:129 returning priority 51
[g092:1207223] pml_ucx.c:114 Pack remote worker address, size 238
[g092:1207223] pml_ucx.c:114 Pack local worker address, size 374
[g092:1207223] pml_ucx.c:351 created ucp context 0x768520, worker 0x7d7f20
[g092:1207223] pml_ucx_component.c:129 returning priority 51
[g090:1261043] pml_ucx.c:182 Got proc 0 address, size 374
[g090:1261043] pml_ucx.c:411 connecting to proc. 0
[g090:1261044] pml_ucx.c:182 Got proc 1 address, size 374
[g090:1261044] pml_ucx.c:411 connecting to proc. 1
[g091:1549666] pml_ucx.c:182 Got proc 3 address, size 374
[g091:1549666] pml_ucx.c:411 connecting to proc. 3
[g092:1207223] pml_ucx.c:182 Got proc 4 address, size 374
[g092:1207223] pml_ucx.c:411 connecting to proc. 4
[g093:1949867] pml_ucx.c:182 Got proc 5 address, size 374
[g093:1949867] pml_ucx.c:411 connecting to proc. 5
[g091:1549665] pml_ucx.c:182 Got proc 2 address, size 374
[g091:1549665] pml_ucx.c:411 connecting to proc. 2
[g090:1261043] pml_ucx.c:182 Got proc 1 address, size 374
[g090:1261043] pml_ucx.c:411 connecting to proc. 1
[g090:1261044] pml_ucx.c:182 Got proc 0 address, size 374
[g090:1261044] pml_ucx.c:411 connecting to proc. 0
[g091:1549666] pml_ucx.c:182 Got proc 2 address, size 374
[g091:1549666] pml_ucx.c:411 connecting to proc. 2
[g091:1549665] pml_ucx.c:182 Got proc 3 address, size 374
[g091:1549665] pml_ucx.c:411 connecting to proc. 3
-------------------------------------------------------------
STREAM version $Revision: 1.8 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Total Aggregate Array size = 105000000 (elements)
Total Aggregate Memory per array = 801.1 MiB (= 0.8 GiB).
Total Aggregate memory required = 2403.3 MiB (= 2.3 GiB).
Data is distributed across 6 MPI ranks
Array size per MPI rank = 17500000 (elements)
Memory per array per MPI rank = 133.5 MiB (= 0.1 GiB).
Total memory per MPI rank = 400.5 MiB (= 0.4 GiB).
-------------------------------------------------------------
Each kernel will be executed 10 times.
The *best* time for each kernel (excluding the first iteration)
will be used to compute the reported bandwidth.
The SCALAR value used for this run is 0.420000
-------------------------------------------------------------
Your timer granularity/precision appears to be 1 microseconds.
[g091:1549666] pml_ucx.c:182 Got proc 0 address, size 238
[g091:1549666] pml_ucx.c:411 connecting to proc. 0
[g093:1949867] pml_ucx.c:182 Got proc 0 address, size 238
[g093:1949867] pml_ucx.c:411 connecting to proc. 0
[g092:1207223] pml_ucx.c:182 Got proc 0 address, size 238
[g092:1207223] pml_ucx.c:411 connecting to proc. 0
[g091:1549665] pml_ucx.c:182 Got proc 0 address, size 238
[g091:1549665] pml_ucx.c:411 connecting to proc. 0
Each test below will take on the order of 14543 microseconds.
(= 14543 timer ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 timer ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
[g090:1261043] pml_ucx.c:182 Got proc 2 address, size 238
[g090:1261043] pml_ucx.c:411 connecting to proc. 2
[g090:1261043] pml_ucx.c:182 Got proc 3 address, size 238
[g090:1261043] pml_ucx.c:411 connecting to proc. 3
[g090:1261043] pml_ucx.c:182 Got proc 4 address, size 238
[g090:1261043] pml_ucx.c:411 connecting to proc. 4
[g090:1261043] pml_ucx.c:182 Got proc 5 address, size 238
[g090:1261043] pml_ucx.c:411 connecting to proc. 5
[g093:1949867] pml_ucx.c:182 Got proc 4 address, size 238
[g093:1949867] pml_ucx.c:411 connecting to proc. 4
Function Best Rate MB/s Avg time Min time Max time
Copy: 83323.4 0.029720 0.020162 0.048685
Scale: 81398.2 0.028492 0.020639 0.043052
Add: 86355.2 0.040145 0.029182 0.049721
Triad: 86804.4 0.039043 0.029031 0.047365
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------
[g090:1261043] common_ucx.c:440 disconnecting from rank 0
[g090:1261044] common_ucx.c:440 disconnecting from rank 1
[g093:1949867] common_ucx.c:440 disconnecting from rank 5
[g092:1207223] common_ucx.c:440 disconnecting from rank 4
[g091:1549666] common_ucx.c:440 disconnecting from rank 2
[g091:1549665] common_ucx.c:440 disconnecting from rank 0
[g090:1261043] common_ucx.c:440 disconnecting from rank 1
[g090:1261043] common_ucx.c:440 disconnecting from rank 2
[g090:1261044] common_ucx.c:440 disconnecting from rank 0
[g090:1261044] common_ucx.c:404 waiting for 0 disconnect requests
[g090:1261043] common_ucx.c:404 waiting for 1 disconnect requests
[g090:1261043] common_ucx.c:440 disconnecting from rank 3
[g090:1261043] common_ucx.c:404 waiting for 1 disconnect requests
[g090:1261043] common_ucx.c:440 disconnecting from rank 4
[g090:1261043] common_ucx.c:404 waiting for 1 disconnect requests
[g090:1261043] common_ucx.c:440 disconnecting from rank 5
[g090:1261043] common_ucx.c:404 waiting for 1 disconnect requests
[g090:1261043] common_ucx.c:404 waiting for 0 disconnect requests
[g091:1549665] common_ucx.c:404 waiting for 1 disconnect requests
[g092:1207223] common_ucx.c:440 disconnecting from rank 0
[g092:1207223] common_ucx.c:404 waiting for 1 disconnect requests
[g092:1207223] common_ucx.c:404 waiting for 0 disconnect requests
[g091:1549665] common_ucx.c:440 disconnecting from rank 2
[g091:1549665] common_ucx.c:440 disconnecting from rank 3
[g093:1949867] common_ucx.c:440 disconnecting from rank 0
[g093:1949867] common_ucx.c:404 waiting for 1 disconnect requests
[g093:1949867] common_ucx.c:440 disconnecting from rank 4
[g093:1949867] common_ucx.c:404 waiting for 1 disconnect requests
[g093:1949867] common_ucx.c:404 waiting for 0 disconnect requests
[g091:1549665] common_ucx.c:404 waiting for 0 disconnect requests
[g091:1549666] common_ucx.c:440 disconnecting from rank 3
[g091:1549666] common_ucx.c:440 disconnecting from rank 0
[g091:1549666] common_ucx.c:404 waiting for 1 disconnect requests
[g091:1549666] common_ucx.c:404 waiting for 0 disconnect requests
[g090:1261044] pml_ucx.c:367 mca_pml_ucx_cleanup
[g090:1261043] pml_ucx.c:367 mca_pml_ucx_cleanup
[g091:1549665] pml_ucx.c:367 mca_pml_ucx_cleanup
[g093:1949867] pml_ucx.c:367 mca_pml_ucx_cleanup
[g091:1549666] pml_ucx.c:367 mca_pml_ucx_cleanup
[g092:1207223] pml_ucx.c:367 mca_pml_ucx_cleanup
[g090:1261044] pml_ucx.c:268 mca_pml_ucx_close
[g091:1549665] pml_ucx.c:268 mca_pml_ucx_close
[g091:1549666] pml_ucx.c:268 mca_pml_ucx_close
[g090:1261043] pml_ucx.c:268 mca_pml_ucx_close
[g092:1207223] pml_ucx.c:268 mca_pml_ucx_close
[g093:1949867] pml_ucx.c:268 mca_pml_ucx_close
@RobbieTheK i don't see the error "PML ucx cannot be selected" here, seems UCX is used. also, could you pls run "ucx_info -v" and post the output?
We don't have ucx installed, isn't it available in OpenMPI 4.1.1? From the verbose output does this apply? UCX version 1.11.0
ucx_info -v
-bash: ucx_info: command not found
Here's what I get with OSU's benchmark:
mpirun -mca pml_ucx_verbose 999 --mca opal_warn_on_missing_libcuda 0 --mca orte_base_help_aggregate 0 --host xxx,xxx --oversubscribe -n 2 ./osu_latency_mp
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: xxx
Local device: mlx5_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: xxx
Local device: mlx5_0
--------------------------------------------------------------------------
[g171:2707658] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g171:2707659] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.0
[g171:2707658] common_ucx.c:304 posix/memory: did not match transport list
[g171:2707658] common_ucx.c:304 sysv/memory: did not match transport list
[g171:2707658] common_ucx.c:304 self/memory0: did not match transport list
[g171:2707658] common_ucx.c:304 tcp/lo: did not match transport list
[g171:2707658] common_ucx.c:304 tcp/eth0: did not match transport list
[g171:2707658] common_ucx.c:304 tcp/ib0: did not match transport list
[g171:2707658] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g171:2707658] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g171:2707658] common_ucx.c:311 support level is transports and devices
[g171:2707658] pml_ucx.c:289 mca_pml_ucx_init
[g171:2707659] common_ucx.c:304 posix/memory: did not match transport list
[g171:2707659] common_ucx.c:304 sysv/memory: did not match transport list
[g171:2707659] common_ucx.c:304 self/memory0: did not match transport list
[g171:2707659] common_ucx.c:304 tcp/lo: did not match transport list
[g171:2707659] common_ucx.c:304 tcp/eth0: did not match transport list
[g171:2707659] common_ucx.c:304 tcp/ib0: did not match transport list
[g171:2707659] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[g171:2707659] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[g171:2707659] common_ucx.c:311 support level is transports and devices
[g171:2707659] pml_ucx.c:289 mca_pml_ucx_init
[g171:2707658] pml_ucx.c:114 Pack remote worker address, size 238
[g171:2707658] pml_ucx.c:114 Pack local worker address, size 374
[g171:2707658] pml_ucx.c:351 created ucp context 0x78dcd0, worker 0x7fce80
[g171:2707658] pml_ucx_component.c:129 returning priority 51
[g171:2707659] pml_ucx.c:114 Pack remote worker address, size 238
[g171:2707659] pml_ucx.c:114 Pack local worker address, size 374
[g171:2707659] pml_ucx.c:351 created ucp context 0x78c950, worker 0x7fc0b0
[g171:2707659] pml_ucx_component.c:129 returning priority 51
[g171:2707658] pml_ucx.c:182 Got proc 0 address, size 374
[g171:2707658] pml_ucx.c:411 connecting to proc. 0
[g171:2707659] pml_ucx.c:182 Got proc 1 address, size 374
[g171:2707659] pml_ucx.c:411 connecting to proc. 1
[g171:2707658] pml_ucx.c:182 Got proc 1 address, size 374
[g171:2707658] pml_ucx.c:411 connecting to proc. 1
[g171:2707659] pml_ucx.c:182 Got proc 0 address, size 374
[g171:2707659] pml_ucx.c:411 connecting to proc. 0
# Number of forked processes in sender: 2
# Number of forked processes in receiver: 2
# OSU MPI Multi-process Latency Test v6.1
# Size Latency (us)
0 0.27
1 0.27
2 0.27
4 0.27
8 0.27
16 0.27
32 0.28
64 0.30
128 0.36
256 0.39
512 0.45
1024 0.56
2048 0.74
4096 1.09
8192 1.78
16384 2.89
32768 4.61
65536 8.07
131072 14.87
262144 11.70
524288 21.96
1048576 46.74
2097152 133.04
4194304 293.98
[g171:2707659] common_ucx.c:440 disconnecting from rank 1
[g171:2707658] common_ucx.c:440 disconnecting from rank 0
[g171:2707658] common_ucx.c:440 disconnecting from rank 1
[g171:2707658] common_ucx.c:404 waiting for 0 disconnect requests
[g171:2707659] common_ucx.c:440 disconnecting from rank 0
[g171:2707659] common_ucx.c:404 waiting for 0 disconnect requests
[g171:2707658] pml_ucx.c:367 mca_pml_ucx_cleanup
[g171:2707659] pml_ucx.c:367 mca_pml_ucx_cleanup
[g171:2707659] pml_ucx.c:268 mca_pml_ucx_close
[g171:2707658] pml_ucx.c:268 mca_pml_ucx_close
OpenMPI was probably build with custom UCX path, need to run ucx_info from there Anyhow, according to this output UCX is being used correctly
OK so my questions are why the WARNING: There was an error initializing an OpenFabrics device and segfault when using --mca btl openib and does Device vendor ID: 0x02c9 need to be added to the .ini file?
How would I find ucx here?
openmpi-4.1.1_ucx_cuda_11.0.3_support]$ find . -name *ucx*
./lib/openmpi/mca_sshmem_ucx.so
./lib/openmpi/mca_spml_ucx.so
./lib/openmpi/mca_osc_ucx.la
./lib/openmpi/mca_sshmem_ucx.la
./lib/openmpi/mca_osc_ucx.so
./lib/openmpi/mca_pml_ucx.la
./lib/openmpi/mca_atomic_ucx.so
./lib/openmpi/mca_atomic_ucx.la
./lib/openmpi/mca_spml_ucx.la
./lib/openmpi/mca_pml_ucx.so
./lib/libmca_common_ucx.la
./lib/libmca_common_ucx.so.40
./lib/libmca_common_ucx.so.40.30.1
./lib/libmca_common_ucx.so
OK so my questions are why the
WARNING: There was an error initializing an OpenFabrics deviceand segfault when using--mca btl openiband does Device vendor ID: 0x02c9 need to be added to the .ini file?
I'd advise to disable btl/openib component since it was deprecated and replaced by ucx: add -mca btl ^openib to mpirun.
How would I find ucx here?
ldd ./lib/openmpi/mca_pml_ucx.so|grep libucp
I'd advise to disable btl/openib component since it was deprecated and replaced by ucx: add
-mca btl ^openibto mpirun
Thanks was aware of that just was suggesting a more useful message/log rather than segfault.
Does anyone know of a way to better use these benchmarks to find if there is a performance difference, e.g., if one's faster than the other? I get very similar results using STREAMS and OSU's micro benchmarks.
also:
ldd ./lib/openmpi/mca_pml_ucx.so|grep libucp
libucp.so.0 => /burg/opt/ucx_cuda_11.0.3_support/lib/libucp.so.0 (0x0000155554440000)
But is there a way to run ucx standalone when it's compiled in OpenMPI?
@RobbieTheK i don't see the error "PML ucx cannot be selected" here, seems UCX is used.
This error happens when I use the OSU Multi-threaded Latency Test and likely, OpenMPI was not compiled with OCX support. Can I make a suggestion to improve the warning to mention that?
I think just ucx_info -v should show the config options
— Tomislav Janjusic
From: RobbieTheK @.> Sent: Friday, September 23, 2022 4:08:56 PM To: open-mpi/ompi @.> Cc: Tomislav Janjusic @.>; Team mention @.> Subject: Re: [open-mpi/ompi] Device vendor ID: 0x02c9, part ID: 4123, WARNING: No preset parameters were found for the device that Open MPI detected (Issue #10841)
UCX built with multithreaded support?
Ah good question was done before my time. Is there a way to use -x UCX_LOG_LEVEL=data to find out?
— Reply to this email directly, view it on GitHubhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-mpi%2Fompi%2Fissues%2F10841%23issuecomment-1256680685&data=05%7C01%7Ctomislavj%40nvidia.com%7C09b0444d1cbd426b4e4b08da9da7d406%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637995641385650409%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FaRl9gxwIbFleS8j4j2zDRB50OdrBwzuXh1OnKjxChY%3D&reserved=0, or unsubscribehttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAZCSJDN4DGFQ7B7OI5N35DV7YL6RANCNFSM6AAAAAAQUBJVX4&data=05%7C01%7Ctomislavj%40nvidia.com%7C09b0444d1cbd426b4e4b08da9da7d406%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637995641385650409%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0gXC4UPL1djLafL4Aviw5e%2BcRhP2UJl1c%2FfK7lp%2BUdU%3D&reserved=0. You are receiving this because you are on a team that was mentioned.Message ID: @.***>
another one.
WARNING: No preset parameters were found for the device that Open MPI detected: Device name: irdma0 Device vendor ID: 0x8086 Device vendor part ID: 5522
