Add support for 8k tokens with Claude 3.5 Sonnet

[WilliamAGH]

Issue

Claude 3.5 Sonnet now supports a maximum output/response token length of 8,192 tokens (from 4,096 currently).

To support 8k tokens, we need to add the header "anthropic-beta": "max-tokens-3-5-sonnet-2024-07-15" to our API calls:

[flamerged]

Issue

Claude 3.5 Sonnet now supports a maximum output/response token length of 8,192 tokens (from 4,096 currently).

To support 8k tokens, we need to add the header "anthropic-beta": "max-tokens-3-5-sonnet-2024-07-15" to our API calls:

How did you get past this:

[pascalandy]

official docs about the feat: https://docs.anthropic.com/en/docs/about-claude/models#model-comparison

[bitnom]

did anybody make a PR to fix?

[paul-gauthier]

Thanks for trying aider and filing this issue.

Aider is already able to receive unlimited output from Sonnet. See the article below. This capability was introduced back in v0.41.0.

https://aider.chat/2024/07/01/sonnet-not-lazy.html

That said, it would be beneficial to support the new 8k output limit, mainly to somewhat reduce the cost and latency of longer LLM responses.

[paul-gauthier]

I added support for 8k output tokens.

The change is available in the main branch. You can get it by installing the latest version from github:

python -m pip install --upgrade git+https://github.com/paul-gauthier/aider.git

If you have a chance to try it, let me know if it works better for you.

[paul-gauthier]

I'm going to close this issue for now, but feel free to add a comment here and I will re-open or file a new issue any time.

[berwinsingh]

I am trying to do this on the AWS Bedrock hosted version of 3.5 Sonnet. However, to my surprise I found that it isn't really updated all that much meaning if I pass this parameter I get an error of extra headers not allowed.

Then doing some searching I found that Sonnet on bedrock hasn't been updated since it was launched i.e., June 2024 which is really surprising. Any idea if Amazon or Anthropic will update the models?

[Prince-Mendiratta]

facing same issue as @berwinsingh

[jackyliang]

Is this on by default, or do I need to point Cursor to use a specific model/API?

[WConsulting]

Is this available in the Claude 3.5 Sonnet chat? Seems like it is still being limited 4k tokens or 15,000 characters.

When looking at the models documentation it says 8k tokens output.

https://docs.anthropic.com/en/docs/about-claude/models#model-comparison-table

[paul-gauthier]

Aider is using 8k output tokens for sonnet:

aider --sonnet --verbose

...

Model metadata:
{
    "key": "claude-3-5-sonnet-20241022",
    "max_tokens": 8192,

...