Emma Qiao

Hi @jperez999 do you still have problems on this?

We have a notebook for generating keyset, please refer to https://github.com/NVIDIA-Merlin/HugeCTR/blob/main/notebooks/embedding_training_cache_example.ipynb

We are now using cudf installed from upstream merlin-base image.

/ok to test

Hi @SeekPoint, thanks for the finding. Could you give more information: How did you build image 'merlinbase'? Which merlin branch was used for 'dockerfile.merlin' and 'dockerfile.ctr'? I can't reproduce it...

Hi @SeekPoint ,Sorry, I still can't reproduce it although I checkout merlin v23.08.00 to build as the commands which you provided. Could you check/provide these info: 1. Did these lines(https://github.com/NVIDIA-Merlin/Merlin/blob/release-23.08/docker/dockerfile.ctr#L57-L58)...

/ok to test

/ok to test

Hi @SeekPoint, Thanks for trying hugectr. Registry 'nvcr.io/nvstaging' can only be used internally. The merlin-base could be built from dockerfile.merlin(same path as dockerfile.ctr). Then pass the --build-arg BASE_IMAGE=$image_tag_built_from_dockerfile.merlin when docker...

Hi RaghavendraChari, I can't reproduce this error on 2 node in our cluster even I built the image from 'training_results_v3.0' repo. Could you provide the detail reproduce steps? How did...