[Resolver] Pass retry logic environment variables in GitHub Action

[FabianHertwig]

What problem or use case are you trying to solve?

When using the OpenHands Resolver with Claude I run into rate limit errors most of the time.

Describe the UX of the solution you'd like

I would like to be able to pass the rate limit configuration to the GitHub Action

Do you have thoughts on the technical implementation?

I would like to add

  LLM_RETRY_MIN_WAIT: 60
  LLM_RETRY_MAX_WAIT: 600

to my workflow like in the example below:

jobs:
  call-openhands-resolver:
    if: |
      ${{
        github.event.label.name == 'fix-me' ||
        (github.event_name == 'issue_comment' &&
        startsWith(github.event.comment.body, vars.OPENHANDS_MACRO || '@openhands-agent') &&
        (github.event.comment.author_association == 'OWNER' || github.event.comment.author_association == 'COLLABORATOR' || github.event.comment.author_association == 'MEMBER'))
      }}
    uses: All-Hands-AI/OpenHands/.github/workflows/openhands-resolver.yml@main
    with:
      macro: ${{ vars.OPENHANDS_MACRO || '@openhands-agent' }}
      max_iterations: 50
      LLM_RETRY_MIN_WAIT: 60
      LLM_RETRY_MAX_WAIT: 600
    secrets:
      PAT_TOKEN: ${{ secrets.PAT_TOKEN }}
      PAT_USERNAME: ${{ secrets.PAT_USERNAME }}
      LLM_MODEL: ${{ secrets.LLM_MODEL }}
      LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
      LLM_BASE_URL: ${{ secrets.LLM_BASE_URL }}

Ideally the other configuration values are also supported

LLM_NUM_RETRIES (Default of 8)
LLM_RETRY_MIN_WAIT (Default of 15 seconds)
LLM_RETRY_MAX_WAIT (Default of 120 seconds)
LLM_RETRY_MULTIPLIER (Default of 2)

I think in the All-Hands-AI/OpenHands/.github/workflows/openhands-resolver.yml they must then be added as inputs and passed on with the other env variables.

https://github.com/All-Hands-AI/OpenHands/blob/104f52bcdd4859c68d241a00d4f46d0bad658572/.github/workflows/openhands-resolver.yml#L3-L24

https://github.com/All-Hands-AI/OpenHands/blob/104f52bcdd4859c68d241a00d4f46d0bad658572/.github/workflows/openhands-resolver.yml#L150-L164

[FabianHertwig]

This might be related to https://github.com/All-Hands-AI/OpenHands/issues/5030

[FabianHertwig]

I have added some funds to my Anthropic account to get into the Tier 2. Now it runs much better :).

Maybe there should be a hint in the docs where also the rate limit configuration is described that the Anthropic Tier 1 is really not sufficient to run OpenHands. I did not even know that the rate limit is tied to tiers.

[mamoodi]

That's interesting. I did not know about that personally. @enyst is that true? That Tier1 is not sufficient?

[enyst]

Yes, I think Fabian is correct. I do remember that Anthropic rate limits started by being atrocious, and practically unusable. I had to turn our config settings significantly up, and so did other people in issues around here.

(We also changed our defaults multiple times, but I don't think we changed them so much)

Anthropic has the tiers in this table. 50 requests per minute - I don't know if those changed in time but anyway for the resolver or other headless run... it still sounds really low.

And the worse might be the tokens per minute one. We don't handle that, so the workaround is to estimate it by turning upper the other settings.

The documentation issue is IMHO different than the fact that it would be nice if the resolver allows it.

[FabianHertwig]

The jump in requests per minute from 50 in tier one to 1000 in tier 2 is quite big. Tier 3 and 4 "only" double it to 2000 and 4000.

So with tier 1 I got a lot of rate limit errors. With tier 2 OpenHands runs quite well and only sometimes runs into rate limits.

The resolution to my original issue is really to get into tier 2. Therefore I think that should get mentioned in the getting started documentation.

Being able to pass the rate limit configuration is probably nice to have, but not a good solution to the issue I had. If I would have been able to do that, then probably OpenHands would have been super slow.

[ghost]

Thanks for sharing your experience! I completely agree that the jump from tier 1 to tier 2 can be significant in terms of UX. However, there are a few factors worth considering here:

1. Tier 1 Can Still Attract Potential Customers: Even though tier 1 may seem restrictive, it can still attract many users with low budgets or for smaller-scale applications. A lot of startups, hobbyists, or smaller businesses may not need the high throughput provided by tiers 2-4, and they might prioritize cost over speed.
2. Rate Limit Workarounds: While passing the rate limit configuration could potentially make OpenHands Resolver slower, there are other ways to mitigate rate limits in tier 1. For example, strategic scheduling of requests, intelligent batching, or implementing retries with exponential backoff strategies can help avoid hitting rate limits frequently. This could allow users in tier 1 to still have a reasonable experience without immediately jumping to tier 2.
3. Scalability and Flexibility: I do agree that providing more flexibility in rate limit configuration would be nice, but it’s also important to remember that OpenHands Resolver is built to scale efficiently by running as a GitHub Actions. For users on tier 1, optimizing the system to work within those limits is a good way to keep the service both cost-effective and functional. It’s about balancing performance with budget constraints.

[FabianHertwig]

That is true. Though to get to tier 2 you only have to add 40$.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions[bot]]

This issue was closed because it has been stalled for over 30 days with no activity.