unitxt
unitxt copied to clipboard
IBM
🦄 Unitxt: a python library for getting data fired up and set for training and evaluation
InferenceEngine abstraction can make model based metrics model invariant and updated with ease - [ ] Detector - [ ] Safety - [ ] Regard A great example is the...
Those metrics cannot be loaded trough huggingface load dataset until re implementated as unitxt metrics.
For example, Unitxt relies on the following fields being part of the instance: {str} 'recipe_metadata' {str} 'data_classification_policy' However, we have 2 operators that delete them from the stream: - SelectFields()...
with proper deprecation flag for CastFIelds
prototype: ```python class WebModuleOperator(FieldOperator): source_urls: List[str] # all downloaded to tempdir function: str # imported from tempdir when set hash: str # used to make sure the files hasnt changed...
we need a way for lm-eval to import user packages, so the metric get registered.
OLMES includes the following elements, justified in detail above: - [ ] Use test set when available, otherwise validation. Sample 1000 instances if more than 1500 - [x] Use specified,...
To be shorter and more consistent.
