NeMo-Curator icon indicating copy to clipboard operation
NeMo-Curator copied to clipboard

Semantic Dedup doesn't work with UCX

Open praateekmahajan opened this issue 4 months ago • 1 comments

Describe the bug

Semantic Dedup often gets stuck at the state when we call semantic_cluster_dedup.extract_dedup_data.

Steps/Code to reproduce bug

Run semantic dedup when the client = get_client(device_type='gpu', protocol='ucx')

Environment overview

Tried on cudf-cu12=24.8.* and cudf-cu12==24.10.a*

Succeeds when protocol='tcp'

praateekmahajan avatar Oct 08 '24 00:10 praateekmahajan