NeMo-Curator icon indicating copy to clipboard operation
NeMo-Curator copied to clipboard

Published 20 hours ago • verified publisher icon NVIDIA

Deprecate `max_text_bytes_per_part`

Open sarahyurick opened this issue 1 year ago • 1 comments
trafficstars

We have long strings support in cuDF now, so we can deprecate the max_text_bytes_per_part parameter.

Related:

  • https://github.com/NVIDIA/NeMo-Curator/pull/77
  • https://github.com/NVIDIA/NeMo-Curator/issues/233
  • https://github.com/NVIDIA/NeMo-Curator/pull/314

sarahyurick avatar Oct 28 '24 20:10 sarahyurick

Will work on this after https://github.com/NVIDIA/NeMo-Curator/pull/316 is resolved.

sarahyurick avatar Oct 28 '24 22:10 sarahyurick

Closing as no longer relevant after the Ray refactor.

sarahyurick avatar Oct 13 '25 16:10 sarahyurick