Buffers flagged as P2P can't be read from / written by host via DMA

[moazin]

Buffers that are P2P are painfully slow to read and write currently. This is because the transfer is not happening via the DMA.

Please see the chart below.

To make life easier, I am providing a program that I used to get these results. I wrote it inside the Vitis Accel Repository to use some of the Makefiles and common includes there. Here are the instructions to make it run.

1. Clone the Vitis_Accel_Examples repository.
2. Create a folder inside the root of the repository titled bandwidth_experiments.
3. Unzip my attached ZIP file inside the bandwidth_experiments folder and extract it in a folder named bandwidth_test_for_xilinx.
4. The path should look like Vitis_Accel_Examples/bandwidth_experiments/bandwidth_test_for_xilinx/src/host.cpp.
5. Build the host program and the kernel and run the host program giving it the path to xclbin of the adder kernel.

bandwidth_test_for_xilinx.zip

@maxzhen @uday610

[maxzhen]

We have taken a look at this issue and the current thinking is to use xclReadBO and xclWriteBO to move data to/from P2P buffer via DMA. But, we have not got cycles to change the driver to support it. I believe that we need to change the code in xocl_pread_bo_ioctl() and xocl_pwrite_bo_ioctl() to make sure, for P2P BO, we go through xocl_migrate_unmgd(), instead of copy_to_user/copy_from_user. Maybe @moazin , you can make the change and file a pull request? Or we will see if we could find some time to to it in next release (2023.1).

[ndcontini]

Any chance we can see this fix in an upcoming release? I have experienced this in some of my experiments as well.