MIOpen
MIOpen copied to clipboard
ROCmSoftwarePlatform
MIOpen(HIP): Warning [SQLiteBase] Missing system database file: gfx90878.kdb
I get the above warning however, a file called gfx90878.db is available. I have also copied this file with the name gfx90878.kdb to no avail.
Please advise.
@jamaliki are you using Pytorch wheel? If so the KDB needs to be embedded and cannot be copied over. If you are building MIOpen from code, the KDB file is in git-lfs state and for example this one: https://github.com/ROCmSoftwarePlatform/MIOpen/blob/develop/src/kernels/gfx90878.kdb
needs to be embedded and cannot be copied over
Why is this the case. We can trace its system calls and see its looking for the kdb in a folder like: /usr/lib/python3.10/site-packages/torch/share/miopen/db/. However even if the kdb is in the exact location its looking for it still fails.
IF you're using a PyTorch wheel, this issue should be addressed once we move to using non-embedded build of MIOpen. The rough timeline for that change to make it into upstream PyTorch is by ROCm5.5.
Why is this the case
As I understand, it is because when building an embedded version of MIOpen (which is what the PyTorch wheels currently use), it disables file I/O so it can't read kdb files from disk.
Found a work around that works for me (on arch). Steal the binary from the miopen-hip package (need the arch4edu repo enabled). Like so cp /opt/rocm/miopen/lib/libMIOpen.so $(python -m site --user-site)/torch/lib/libMIOpen.so. This will get rid of the warning message and let it store the kernels it compiles, which makes many things much faster.
@jamaliki are you using Pytorch wheel? If so the KDB needs to be embedded and cannot be copied over. If you are building MIOpen from code, the KDB file is in git-lfs state and for example this one: https://github.com/ROCmSoftwarePlatform/MIOpen/blob/develop/src/kernels/gfx90878.kdb
I am using the Pytorch wheel and I think this is what is making things very slow. What is your suggestion for me to do instead?
@jamaliki Please read this: https://github.com/ROCmSoftwarePlatform/MIOpen/issues/1391#issuecomment-1239897584. Maybe I am wrong but possibly your question should be asked at https://github.com/ROCmSoftwarePlatform/pytorch.
@jamaliki Has this issue been resolved? If so, please close ticket. Thanks!
