NeMo-Curator icon indicating copy to clipboard operation
NeMo-Curator copied to clipboard

[FEA] Align the character pruning vs sequence length based pruning for our models.

Open VibhuJawa opened this issue 1 year ago • 0 comments
trafficstars

Is your feature request related to a problem? Please describe.

We need to align the character pruning vs sequence length based pruning for our models. We need to ensure our max_char defaults are lines up and only do it for models that have it built in while training

Additional context https://github.com/NVIDIA/NeMo-Curator/pull/168#discussion_r1723538420

VibhuJawa avatar Aug 20 '24 19:08 VibhuJawa