NeMo-Curator
NeMo-Curator copied to clipboard
Add classifier CLI script tests
trafficstars
Loosely modeled after the NeMo setup:
- https://github.com/NVIDIA/NeMo/tree/main/tests/functional_tests
- https://github.com/NVIDIA/NeMo/blob/main/.github/workflows/cicd-main-e2e-tests.yml
TODO:
- [x] aegis_classifier_inference
- [x] content_type_classifier_inference
- [x] domain_classifier_inference
- [x] fineweb_edu_classifier_inference
- [x] fineweb_mixtral_edu_classifier_inference
- [x] fineweb_nemotron_edu_classifier_inference
- [x] instruction_data_guard_classifier_inference
- [x] multilingual_domain_classifier_inference
- [x] prompt_task_complexity_classifier_inference
- [x] quality_classifier_inference