eval-dev-quality icon indicating copy to clipboard operation
eval-dev-quality copied to clipboard

Rework how we allow to set (query) attributes of models

Open zimmski opened this issue 11 months ago • 0 comments

Please help us keep the number of duplicated issues small. We kindly ask you to add your input to the appropriate issue or PR in case your feature idea is already being tracked.

Problem

OpenAI's o3-mini requires to set reasoning_effort to either low, medium (default) or high.

Solution (optional)

The current implementation allows to set the attribute over the CLI option: --model custom-openai/o3-mini-2025-01-31/reasoning_effort=high. However this is kind of a hack. Find a better way UX and code-wise.

Tasks:

  • [x] Do the hack-implementation: We use "@" because ":" is taken by model names already.
  • [ ] Add at least a test for the hack: check that the full ID is used in the logs and result path directories
  • [ ] Revamp how we hold model IDs e.g. in llm.Model we now have "id" and "modelID" because the first includes even attributes that are nice to see in the logs/result-directories.
  • [ ] TODO

zimmski avatar Feb 01 '25 22:02 zimmski