openucx
UCC calling HCOLL in version 1.4.0
Hello,
we are running some OpenMPI/HPCX-MPI programs on a cluster using ucc/1.4.0. In this OpenMPI installation, UCC has the highest priority, but in some configurations an error related to some unsupported hcoll algorithm appears. To avoid this error we are considering disabling HCOLL communication and forcing UCC with:
export OMPI_MCA_coll_hcoll_enable=0
export OMPI_MCA_coll_ucc_enable=1
but I am not aware of potential consequences of this approach.
Indeed, from the call stack and the -mca coll_ucc_verbose output, we see that UCC is the main library used for collective communications, but -mca coll_hcoll_verbose shows that HCOLL is still used in the following communication:
../../../../../ompi/mca/coll/hcoll/coll_hcoll_rte.c:230 - send_nb() PML_ISEND: dest = 15: buf = 0x7fffe0f12060: size = 8: comm = 0x153c12d14e50
This seems to be related to some metadata, given the small buffer size? Further, by using nsys we noticed from the sampling points that ucc library calls hcoll in some bcast operations (hcoll_ml_progress, picture below). Is this behavior expected, and does it depend on the ucc version?
