NeMo-Curator
NeMo-Curator copied to clipboard
[FEA] Align the character pruning vs sequence length based pruning for our models.
trafficstars
Is your feature request related to a problem? Please describe.
We need to align the character pruning vs sequence length based pruning for our models. We need to ensure our max_char defaults are lines up and only do it for models that have it built in while training
Additional context https://github.com/NVIDIA/NeMo-Curator/pull/168#discussion_r1723538420