autolabel
autolabel copied to clipboard
refuel-ai
Label, clean and enrich text datasets with LLMs.
Currently, the dataset accepts the config object and needs this object to create the ground truth labels and run evaluation. However, the dataset should not need the config object and...
**Is your feature request related to a problem? Please describe.** The cache lookup happens by doing string comparison for the prompt. This can be optimized by doing lookups on hash...
following up on https://github.com/refuel-ai/autolabel/issues/509 I'm facing issues getting testing to work - even for the most basic calls it isn't calling my class functions. @DhruvaBansal00 @rajasbansal would appreciate a 2nd...
**Describe the bug** Currently, the metrics computed by the labeling agent are not stored in teh dataset. **Expected behavior** We should store the metrics from a labeling run in the...
https://wandb.ai/site/prompts
**Is your feature request related to a problem? Please describe.** Identify context length errors across providers and parse them as a new error type called Context Length error **Describe the...
**Is your feature request related to a problem? Please describe.** So I broke my labels into 1024 tokens, but I'm still getting this error: ``` # dry-run -- this tells...
**Is your feature request related to a problem? Please describe.** It would be nice optionally to have a human-readable log of what happens during `agent.run` This would give more clarity...
This is often helpful as a preliminary first step to understand which LLM does the best on a user's specific task and data
1. Row-level metadata columns: * label disagreement type * turnaround time * explanation column * prompt * confidence score 2. Aggregate metrics: Confusion matrix, quality grouped by label 3. Utility...