ragas icon indicating copy to clipboard operation
ragas copied to clipboard

feat: add run_config concurrency controls for experiments

Open dhyaneesh opened this issue 1 month ago • 2 comments

Issue Link / Problem Description

  • Fixes https://github.com/vibrantlabsai/ragas/issues/2457 (experiment concurrency throttling)

Users running @experiment().arun() couldn’t limit concurrent async tasks to honor provider rate limits (e.g., Azure OpenAI). Unlike evaluate(), there was no RunConfig/max_workers option, so experiment tasks always fired at full concurrency.

Changes Made

  • Thread optional run_config + max_workers through ExperimentWrapper.arun() and the @experiment decorator, reusing ragas.async_utils.as_completed with the resolved worker limit.
  • Add unit tests covering RunConfig-based throttling, explicit overrides, and zero/unlimited coercion.
  • Document the new knobs in docs/concepts/experimentation.md and the RunConfig how-to.

Testing

How to Test

  • [x] Automated tests added/updated
    • uv run pytest tests/unit/test_experiment.py
  • [x] Manual testing steps:
    1. uv run async.py (manual script) before the fix showed run_config keyword errors; after the fix it reports the expected max concurrent values (unlimited, run_config=1, override=3).
    2. uv run pytest tests/unit/test_experiment.py -k run_config_max_workers fails on previous commit, passes now.
    3. make test (full suite) – all tests pass.

References

  • Related issues: https://github.com/vibrantlabsai/ragas/issues/2457
  • Documentation: docs/concepts/experimentation.md, docs/howtos/customizations/_run_config.md
  • External references: none

Screenshots/Examples (if applicable)

N/A – behavior verified via tests + manual script.

dhyaneesh avatar Nov 28 '25 10:11 dhyaneesh

Thanks @dhyaneesh but we're deprecating RunConfig. Will need to think from a fresh perspective as to handling the said config in new architecture.

Will keep this PR open for now.

anistark avatar Nov 28 '25 11:11 anistark

I’m happy to refactor this to match the new architecture or contribute to a new one, once I understand the preferred approach and or any needs.

Thanks for keeping the PR open, I’ll wait for guidance on the next steps.

dhyaneesh avatar Nov 28 '25 11:11 dhyaneesh