NVIDIA
Conda packages
As DALI gains adoption within the Python community it would be helpful to distribute conda packages. Is this in-scope for this project?
To start thinking about this, it would be interesting to know what dependencies this software has and waht compilation environment it requires.
cc @datametrician @jakirkham
Hi, thanks for the question. Thanks to the contribution from IBM we have a conda recipe ready.
Conda packages are maintained and distributed by IBM. For details look here.
IBM conda package is for POWER architecture. Using conda it is still possible to install pip whl and this works in DALI case (even if it is not the most recommended way by the conda community). @mrocklin have you encountered any problems using DALI whl in conda environment?
even if it is not the most recommended way by the conda community
To be clear, this is strongly discouraged in the Conda community (not only not recommend). It raises issues with binary compatibility as wheels frequently vendor all library dependencies within the wheel. So two wheels can supply the same dependencies with different versions (like two BLAS libraries) and this will result in symbol collisions that will crash users' programs. The same problem can happen when adding a wheel in a Conda environment.
Right, I forgot about binary compatibility problems. Libs itself are not a big deal - DALI hashes the lib names and do other magic to prevent symbol clash for the libraries it ships in its whl. But system libraries could be a problem. We are discussing shipping DALI conda package but we are far from the decision now.
Hi, any update on this one? It's been almost two years at the moment since "discussing shipping"
Hi @otaj,
I'm sorry, we don't have any updates we can share now. The DALI conda release is a part of a bigger strategy which details we can't share here.
Has there been any update to this? DALI seems to be a good option for multiprocess data loading for use with JAX and it would be very convenient if shipped as a conda package
EDIT: I see that there is version 1.32.0 available through https://github.com/open-ce/DALI-feedstock
Hi @buchholzmd,
Thank you for reaching out. Indeed, DALI is a part of the Open Cognitive Environment (Open-CE), but the effort is not run by NVIDIA. Regarding the original ask, we still don't have any updates we can share, sorry.