NeMo-Curator
NeMo-Curator copied to clipboard
Added a translation pipeline for ctranslate2 inference
trafficstars
Description
This PR enables ctranslate2 model translation. This will work when CrossFit support for ctranslate2 model is added.(PR)
Usage
python3 NeMo-Curator/examples/ct2_trasnlation_example.py --input-data-dir <inp-dir> --output-data-dir <out-dir> --ct2-model-path <ct2-model-dir> --files-per-partition 1 --input-text-field indic_proc_text --tgt-lang mar_Deva
Checklist
- [ ] I am familiar with the Contributing Guide.
- [ ] New or Existing tests cover these changes.
- [ ] The documentation is up to date with these changes.