sanjari-orb

However I want to be able to specify a suite of evaluation tasks, like [tasks_light.yaml](https://github.com/mosaicml/llm-foundry/blob/v0.3.0/scripts/eval/yamls/tasks_light.yaml) and add sections like ``` # In-context learning evaluation icl_tasks: icl_subset_num_batches: 100 # -1, or...

But how will the evaluator know that the new metric I introduce should only be computed on data corresponding to my new task, and not all the icl tasks? I...

I see, got it. I was patching the evaluator building code but wanted to verify if there was another way or not. Thank you for confirming!

`We're working on some refactors that would make it easier.` Is there any ETA on when the refactoring will be merged btw? It would be great to have support for...

Understood. Thanks a lot!