Use context length from lite llm

[rklasen]

Description

When liteLLM is used as LLM provider, "sane defaults" are used for context window length. This is laudable of course, but may be severely limiting if liteLLM also proxies models with very large context windows like the new Gemini models or the newest OpenAI models, which support up to 1M tokens.

This PR introduces changes to query liteLLM for the model options, and hence supported context length, of the model currently in use. Since this is not known before the model is actually selected, this is done asynchronically.

Test Procedure

I've run the modified Cline extension for some example chats, the context length was updated correctly.

Type of Change

• [x] 🐛 Bug fix (non-breaking change which fixes an issue)
• [x] ✨ New feature (non-breaking change which adds functionality)
• [ ] 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [x] ♻️ Refactor Changes
• [ ] 💅 Cosmetic Changes
• [ ] 📚 Documentation update
• [ ] 🏃 Workflow Changes

Pre-flight Checklist

• [x] Changes are limited to a single feature, bugfix or chore (split larger changes into separate PRs)
• [x] Tests are passing (npm test) and code is formatted and linted (npm run format && npm run lint)
• [x] I have created a changeset using npm run changeset (required for user-facing changes)
• [x] I have reviewed contributor guidelines

Tests don't pass, but only on places I haven't touched.

Screenshots

Before, the default context length was being used:

Afterwards, the values from liteLLM are used:

Additional Notes

---

[!IMPORTANT] This PR updates the LiteLLM integration to dynamically fetch and use the model's context window length, enhancing support for models with large context windows.

• Behavior:
  • LiteLlmHandler in litellm.ts now fetches model info asynchronously to update context window length based on the model in use.
  • Adds getApiStreamUsage() to LiteLlmHandler to return usage details without refetching model info.
  • Updates Task class in task/index.ts to handle model info updates via handleModelInfoUpdate().
• Configuration:
  • Adds onModelInfoUpdate callback to ApiHandlerOptions in api.ts.
  • Updates ApiOptions.tsx to display LiteLLM model info in the UI, using live data if available.
• Misc:
  • Adds https-proxy-agent to package.json for proxy support in API requests.

^{This description was created by for 9ed900c21e63357ff245c76308b9dbc9e49028c3. It will automatically update as commits are pushed.}

[changeset-bot[bot]]

🦋 Changeset detected

Latest commit: db16d2402ce8b3b3dd4f4d264a396493e1a8dd8d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
claude-dev	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[rklasen]

Sorry, synced from the wrong branch. Will fix tomorrow from a different machine.

[spflueger]

This feature would be really cool, then I can funnel everything through litellm and get the right numbers :) Any update on this?

[jorgegarciarey]

Hi @rklasen ,

The endpoint /model/info expects a model ID (hash), but in your screenshot I see a model name claude-3-7-sonnet-US.

I use the LiteLLM model name in Cline, not the Model ID (hash), so, this change would not work.

I think that I should define model_info.id in the config.yml in the same way as model_name. It's correct?

https://docs.litellm.ai/docs/proxy/config_settings

[rklasen]

Hi @rklasen ,

The endpoint /model/info expects a model ID (hash), but in your screenshot I see a model name claude-3-7-sonnet-US.

I use the LiteLLM model name in Cline, not the Model ID (hash), so, this change would not work.

I think that I should define model_info.id in the config.yml in the same way as model_name. It's correct?

https://docs.litellm.ai/docs/proxy/config_settings

Hi, thanks for checking it out!

The changes to litellm.ts would first query for all models (ID and hash) and then matches the ID to the chosen name.

const modelData = data.data.find((model: any) => model.model_name === modelId)

But the model names are currently not used in a drop down menu, maybe I should change that, so that the model name doesn't have to be provided explicitly by the user.

[jorgegarciarey]

@rklasen ,

I've just updated the config.yml by renaming the model ID with the same name as model name and the endpoint works successfully.

model_list:
  # AWS Bedrock
  - model_name: bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0
    litellm_params:
      model: bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0
      aws_access_key_id: os.environ/BEDROCK_AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/BEDROCK_AWS_SECRET_ACCESS_KEY
      aws_session_name: os.environ/BEDROCK_AWS_SESSION_NAME
      aws_role_name: os.environ/BEDROCK_AWS_ROLE_NAME
      aws_region_name: os.environ/BEDROCK_AWS_REGION_NAME
      additional_drop_params: ["temperature"]
    model_info:
      id: bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0

[jorgegarciarey]

@celestial-vault , @pashpashpash , @saoudrizwan , what do you think about this PR? It’s quite interesting for those of us using LiteLLM.

[jorgegarciarey]

@pashpashpash , @celestial-vault , @saoudrizwan , any news? thanks!

[jorgegarciarey]

@dcbartlett , @pashpashpash , @celestial-vault , @saoudrizwan , what do you think of this feature? Could we merge it?

[rklasen]

Let me know if there are still issues left open for me to fix. We could really use this feature.

[bramburn]

@dcbartlett , @pashpashpash , @celestial-vault , @saoudrizwan , what do you think of this feature? Could we merge it?

Unlikely you'll get a response bro.

[valinha]

@rklasen It would be nice if this information could also be used to be able to choose the model from the ui with a dropdown and not having to type the model.

[Tombar]

this is awesome! is there a way for us to test it?

[sammcj]

This would be really useful.

[celestial-vault]

@dcbartlett , @pashpashpash , @celestial-vault , @saoudrizwan , what do you think of this feature? Could we merge it?

Unlikely you'll get a response bro.

oof

[sammcj]

@bramburn Unlikely you'll get a response bro.

Hey I think that's a little unfair.

The Cline devs are really busy and they're doing their best to support the large open source community and most of the time I think they do a great job of this. Remember your priorities might not be the same as the maintainers of software.

It's important that we communicate with empathy and understanding as part of open source communities.

To help gain traction on issues / PRs I'd highly recommend starting or adding to a discussion clearly describing the problem / solution, its potential impact, what you've tested or learned about a possible solution. You could also link to the discussion on their Discord (although discord is an information black hole).