ADIOS2 icon indicating copy to clipboard operation
ADIOS2 copied to clipboard

Published 20 hours ago • verified publisher icon ornladios

Stop ICC and oneapi from using OFI MTL

Open vicentebolea opened this issue 7 months ago • 1 comments

This results in errors since we do not have libfabric backend (other than tcp) in CI, we should use btl=lo in openmpi builds.

Here are the errors:

2023-11-15T01:06:13.8230708Z 1157: 9086916adf67:rank1.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.8346074Z 1157: 9086916adf67:rank1: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.8357749Z 1157: 9086916adf67:rank1: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.8359219Z 1157: 9086916adf67:rank1.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.8375259Z 1157: 9086916adf67:rank1.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.8376611Z 1157: 9086916adf67:rank1: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.8946877Z 1157: 9086916adf67:rank1.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.8953203Z 1157: 9086916adf67:rank1: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.8960434Z 1157: --------------------------------------------------------------------------
2023-11-15T01:06:13.8965226Z 1157: Open MPI failed an OFI Libfabric library call (fi_endpoint).  This is highly
2023-11-15T01:06:13.8970456Z 1157: unusual; your job may behave unpredictably (and/or abort) after this.
2023-11-15T01:06:13.8974792Z 1157: 
2023-11-15T01:06:13.8981616Z 1157:   Local host: 9086916adf67
2023-11-15T01:06:13.8988985Z 1157:   Location: mtl_ofi_component.c:513
2023-11-15T01:06:13.8993021Z 1157:   Error: Invalid argument (22)
2023-11-15T01:06:13.8997318Z 1157: --------------------------------------------------------------------------
2023-11-15T01:06:13.9548832Z 1157: 9086916adf67:rank2.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.9553943Z 1157: 9086916adf67:rank2: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.9558870Z 1157: 9086916adf67:rank2.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.9563846Z 1157: 9086916adf67:rank2: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.9568693Z 1157: 9086916adf67:rank2.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.9573691Z 1157: 9086916adf67:rank2: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.9621726Z 1157: 9086916adf67:rank3.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.9628866Z 1157: 9086916adf67:rank3: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.9640151Z 1157: 9086916adf67:rank3.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.9647504Z 1157: 9086916adf67:rank3: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.9659248Z 1157: 9086916adf67:rank3.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.9666560Z 1157: 9086916adf67:rank3: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.9686826Z 1157: 9086916adf67:rank0.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.9693626Z 1157: 9086916adf67:rank0: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.9705183Z 1157: 9086916adf67:rank0.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.9712351Z 1157: 9086916adf67:rank0: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.9724118Z 1157: 9086916adf67:rank0.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.9731630Z 1157: 9086916adf67:rank0: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.9907083Z 1157: 9086916adf67:rank4.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.9914480Z 1157: 9086916adf67:rank4: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.9921890Z 1157: 9086916adf67:rank4.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.9923254Z 1157: 9086916adf67:rank4: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:13.9924425Z 1157: 9086916adf67:rank4.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set
2023-11-15T01:06:13.9925681Z 1157: 9086916adf67:rank4: PSM3 can't open nic unit: 0 (err=23)
2023-11-15T01:06:14.0055382Z 1157: 9086916adf67:rank0.Test.Engine.BP.WriteReadBZIP2.MPI: Failed to get eth0 (unit 0) cpu set

This can be seen in any recent build here is one example:

https://github.com/ornladios/ADIOS2/actions/runs/6871151390/job/18687822024

vicentebolea avatar Nov 15 '23 01:11 vicentebolea

cc @scottwittenburg

vicentebolea avatar Nov 15 '23 01:11 vicentebolea