When defaulting claude-3 to tiktoken, avoid failed network call in huggingface `from_pretrained`

[cspotcode]

trafficstars

Title

I noticed that every time aider calls main_model.token_count(some_string) where model is 'claude-3-5-sonnet-20240620', the logic in _select_tokenizer falls through to try: tokenizer = Tokenizer.from_pretrained(model) which always makes a failed network request and throws an exception.

It attempts to GET https://huggingface.co/claude-3-5-sonnet-20240620/resolve/main/tokenizer.json which returns a 401 Client error: Invalid username or password. However, the error is not the problem: it's that we shouldn't make the HTTP request at all if our intent is to use tiktoken.

This has a slight performance impact on startup time, and I can't tell why it's helping if it's destined to fail, since the comments say claude should be using tiktoken anyway.

Relevant issues

Type

🐛 Bug Fix / enhancement / optimization

Changes

[REQUIRED] Testing - Attach a screenshot of any new tests passing locall

If UI changes, send a screenshot/GIF of working UI fixes