RuntimeError: Programming Device failed: ENODEV (19) No such device

[mariodruiz]

XRT 2.13 or newer has change the way the host code communicates with the Embedded Run Time (ERT). https://github.com/Xilinx/XRT/blob/2022.1/CHANGELOG.rst#added

This is causing problems when downloading the any of the design targeting two interfaces at the same time. See Python trace on the Appendix below.

The first thing to do if you encounter this problem is to triage your system.

1. Run xbutil --version. An example output shown below

   Version              : 2.13.466
   Branch               : 2022.1
   Hash                 : f5505e402c2ca1ffe45eb6d3a9399b23a0dc8776
   Hash Date            : 2022-04-14 17:45:07
   XOCL                 : 2.13.466, f5505e402c2ca1ffe45eb6d3a9399b23a0dc8776
   XCLMGMT              : 2.13.466, f5505e402c2ca1ffe45eb6d3a9399b23a0dc8776
   

2. Run xbutil validate -d <bdf>.

   The tests should PASS

3. Additionally, you can try to download the xclbin file with xbutil

   xbutil program -d <bdf> -u <xclbin>
   

   You should see the following error

   [XRT] ERROR: CU statistic sysfs node corrupted
   [xbutil] ERROR: Could not program device0000:a6:00.1: Invalid argument
   

Solution

Unfortunately, the current solution is to use an XRT version older than 2.13.x for deployment. You can still build the xclbin with XRT 2.13.x. However, to run this on real hardware, please use an older XRT version. The XRT team is working on a solution.

Appendix

Python trace

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-caf63a7616bb> in <module>
      6 xclbin = 'benchmark.intf0.xilinx_u250_gen3x16_xdma_3_1_202020_1/vnx_benchmark_if0.xclbin'
----> 7 ol = pynq.Overlay(xclbin, device=pynq.Device.devices[1])

/scratch/virt-env/envs/pynq/lib/python3.7/site-packages/pynq/overlay.py in __init__(self, bitfile_name, dtbo, download, ignore_version, device)
    352 
    353         if download:
--> 354             self.download()
    355 
    356         self.__doc__ = _build_docstring(self._ip_map._description,

/scratch/virt-env/envs/pynq/lib/python3.7/site-packages/pynq/overlay.py in download(self, dtbo)
    418                     Clocks.set_pl_clk(i)
    419 
--> 420         super().download(self.parser)
    421         if dtbo:
    422             super().insert_dtbo(dtbo)

/scratch/virt-env/envs/pynq/lib/python3.7/site-packages/pynq/bitstream.py in download(self, parser)
    185 
    186         """
--> 187         self.device.download(self, parser)
    188 
    189     def remove_dtbo(self):

/scratch/virt-env/envs/pynq/lib/python3.7/site-packages/pynq/pl_server/xrt_device.py in download(self, bitstream, parser)
    539         with open(bitstream.bitfile_name, 'rb') as f:
    540             data = f.read()
--> 541         self._xrt_download(data)
    542         super().post_download(bitstream, parser)
    543 

/scratch/virt-env/envs/pynq/lib/python3.7/site-packages/pynq/pl_server/xrt_device.py in _xrt_download(self, data)
    531                 self.contexts = old_contexts
    532                 raise RuntimeError("Programming Device failed: " +
--> 533                                    _format_xrt_error(err))
    534         finally:
    535             xrt.xclUnlockDevice(self.handle)

RuntimeError: Programming Device failed: ENODEV (19) No such device

[mariodruiz]

Fixed in https://github.com/Xilinx/XRT/pull/6744 which will be part of the 2022.2 release

[mariodruiz]

Closing this as the solution is part of XRT 2022.2