NeMo-Curator icon indicating copy to clipboard operation
NeMo-Curator copied to clipboard

Published 20 hours ago • verified publisher icon NVIDIA

Refactor separate_by_metadata and Partition On to use the same code paths.

Open VibhuJawa opened this issue 9 months ago • 0 comments

Is your feature request related to a problem? Please describe.

We are adding partition_on (https://github.com/NVIDIA/NeMo-Curator/pull/519) here which is very similar to separate_by_metadata, we should try to refactor separate_by_metadata and make them follow similar code paths.

VibhuJawa avatar Feb 05 '25 23:02 VibhuJawa