[QST] Install Transformers4Rec on Databricks

[alexanderDoria]

❓ Questions & Help

Details

I was able to follow the tutorial for installing NVTabular on Databricks, and I was wondering if the same set of instructions or similar exists for Transformers4Rec. Seems like there are a couple of approaches for installing Transformers4Rec on Databricks:

1. Build the NVTabular Databricks Docker image and conda install -c nvidia transformers4rec within a notebook
2. Modify the Dockerfile to download Transformers4Rec, in the same way that NVTabular is in README by echo "https://conda.anaconda.org/nvidia/linux-64/nvtabular-0.6.1-py38_0.tar.bz2" >> docker/rapids-spec.txt

I tried approach #1 but had incompatible library issues, and I'm currently trying out option #2. However, I'm concerned with incompatibility issues between Transformers4Rec, NVTabular, and even the Databricks runtime -- so I'm wondering if there's an "ideal" way to use a Transformers4Rec Docker image on Databricks.

[rnyak]

@alexanderDoria thanks for your question. We do not have same set of instructions or similar exists for Transformers4Rec. Did you get any issues with option 2 (modify the docker file)? if yes, please share.

[alexanderDoria]

Hi, I confirmed it worked. I simply added the Transformers4Rec download path (https://anaconda.org/nvidia/transformers4rec/0.1.4/download/noarch/transformers4rec-0.1.4-py_0.tar.bz2) to the rapids-spec.txt file. We'll see how it holds up through usage!