[Feature request] claude-3-5-sonnet-20240620 support

[runekaagaard]

Hi, thanks for a great project i use everyday together with my context management library called hjerne (brain in danish) that - via a django app and some elisp - adds functions, etc. to a context, starts a chatgpt-shell sessions and writes the updated functions back to the correct files ... good stuff!

Remember when everybody used chatgpt-4o and thought that was the schnitzel? Kind of lame when you think about it! </sarcasm>

I know claude-shell exists for emacs but chatgpt-shell is far more stable. Would you be willing to accept a PR that extends the supported models with claude-3-5-sonnet-20240620?

[xenodium]

Hi @runekaagaard

Thanks for reaching out on this!

Would you be willing to accept a PR that extends the supported models with claude-3-5-sonnet-20240620?

I've been meaning to head in this direction. The first step was to create shell-maker. There's a lot more logic that can be offloaded from chatgpt-shell.el. Not too sure if shell-maker is the right destination for this logic just yet.

There are two approaches in terms of user experience:

1. Add more logic to a generic dependency and enable other package mantainers to build additional shells (ie. claude-shell, gemini-shell, etc).
2. A single shell capable of swaping providers (similar to how chatgpt-shell currently swaps version/prompts).

You're referring to 2. right?

While I initally gavitated towards 1. (let other devs to build more shells), I think 2. may be better as a user... The experience (key bindings, commands, etc) remain consistent across any provider (only need to learn the package once).

In terms of whether or not to include other models in this package, it kinda depends on the change itself and if chatgpt-shell lends itself to make it more pluggable. We may need a rough prototype first to see how the codebase would evolve before we can make a call.

Would you be ok with iterating on it until we arrive to a good place? For the prototype, I would focus more on just getting it to work and then we can figure out where to move things to if needed. Does that sound ok to you?

ps. Have you seen this? Not sure if you've seen https://github.com/xenodium/chatgpt-shell/issues/201

edit: typo

[runekaagaard]

Hi, thanks for your thoughtful reply!!

Yeah as a user I personally would totally prefer to just be able to switch models in one single app.

I guess a nice structure would be:

• shell-maker.el: makes shells, also for non LLM usage. Some stuff from chatgpt-shell.el could be moved here.
• llm-shell.el: Previously chatgpt-shell.el. Specific to building an AI interface on top of shell-maker.
• chatgpt.el: customizable url, apikey, header, models, etc. and a request handler for openai API.
• claude.el: customizable url, apikey, header, models, etc. and a request handler for claude API.
• llama.el: etc. etc.

I'm not sure how dall-e and babel would fit on top of that as I've never used them.

As per you suggestion I'm currently hacking away to try and simply make claude work :) Both the request and response format is different than openai.

[xenodium]

Yeah as a user I personally would totally prefer to just be able to switch models in one single app.

👍

llm-shell.el: Previously chatgpt-shell.el. Specific to building an AI interface on top of shell-maker.

llm-shell.el would depend on chatgpt.el, claude.el, llama.el... ? That is, to be able to swap models across all providers, correct?

I'm not sure how dall-e and babel would fit on top of that as I've never used them.

Dall-e can maybe stay on its own for now. The code is pretty isolated, only depends on shell-maker. We can consider bringing it into the family later on if we want to.

babel would fit on top of that as I've never used them.

These can be reworked to use llm-shell.el without much trouble. Will also need to split them out into separate git repos to simplify MELPA management (same for shell-maker). Don't worry about these.

As per you suggestion I'm currently hacking away to try and simply make claude work :) Both the request and response format is different than openai.

Great! Btw, your prolly already spotted shell-maker-logging to turn on some logging, but just in case.

Feel free to share a super rough prototype when you get claude API working (no need to clean up). This gives us an initial gut feel for what's needed.

[runekaagaard]

I got something working here:

https://github.com/runekaagaard/chatgpt-shell-plus-claude/blob/main/claude-shell.el

I still havent figured out how to get streaming responses to work so claude-shell-streaming must be nil. https://docs.anthropic.com/en/api/messages-streaming.

[runekaagaard]

Ah didn't see your last reply, thank you for taking time to reply!

Yes llm-shell would be all the common parts refactored out and then call the correct vendor specific functions in chatgpt, claude, etc.

I noticed logging pretty late in the process, but now I've found it :)

Been thinking more about your two approaches. I guess it depends on how similar the code will be between vendors and models. Right now it's pretty similar, but i guess features like chatgpt functions, claude artifacs, etc. etc. could make a difference in the future. I could see approach 1 working well too, but then you would have a claude buffer, chatgpt buffer. Hmmm, what do you think?

[runekaagaard]

Got streaming working, I think we have a claude mode now :)

EDIT: I had to create custom claude-shell-async-shell-command function. Hmmm. Surely some kind of LLM bridge that mirrors openai's api must exist that would give us ALL vendors/models for free!?!? Thus making claude-shell completely moot :)

[runekaagaard]

Of course MULTIPLE LLM bridges exists, I'm so stupid :)

$ pip install 'litellm[proxy]'
$ litellm --model claude-3-5-sonnet-20240620 --host localhost
INFO:     Uvicorn running on http://localhost:4000 (Press CTRL+C to quit)

(setq chatgpt-shell-api-url-base "http://127.0.0.1:4000")

[xenodium]

Got streaming working, I think we have a claude mode now :)

Nice work!

EDIT: I had to create custom claude-shell-async-shell-command function. Hmmm. Surely some kind of LLM bridge that mirrors openai's api must exist that would give us ALL vendors/models for free!?!? Thus making claude-shell completely moot :)

$ pip install 'litellm[proxy]' $ litellm --model claude-3-5-sonnet-20240620 --host localhost INFO: Uvicorn running on http://localhost:4000 (Press CTRL+C to quit) (setq chatgpt-shell-api-url-base "http://127.0.0.1:4000")

Whoa. This is very cool.

Of course MULTIPLE LLM bridges exists

I'm interested in the prototype changes you made! Are they in the repos you created? When I started this project, there were early ChatGPT Emacs integrations relying on python packages too. While they do the job, their Emacs package installations aren't as straightforward as just package install an go.

[runekaagaard]

Yes the functions i had to change to make it work for claude can be found here: https://github.com/runekaagaard/chatgpt-shell-plus-claude/blob/main/claude-shell.el, the main ones being:

• claude-shell--extract-claude-response
• claude-shell-async-shell-command

[xenodium]

Thanks for this! I'll play with it soon!

[ccqpein]

@runekaagaard Amazing work. Gonna play around it.

[fraricci]

Just wanted to mention that this package is also available and could be merged maybe? https://github.com/arminfriedl/claude-shell

[xenodium]

I'd like to make Claude support happen, but first need some base work. If still keen, please upvote to gauge interest https://github.com/xenodium/chatgpt-shell/issues/244

[xenodium]

Hi @runekaagaard @ccqpein @fraricci @gavinhughes 👋👋

Multi-model support is quietely out of the bag.

I need feedback. Fancy giving Claude support a try?

Set chatgpt-shell-anthropic-key and everything else should work the same 🤞

You need chatgpt-shell v2.0.1 and shell-maker v0.68.1.

[xenodium]

[ccqpein]

@xenodium Amazing work!

I just played around a bit, and here are my thoughts on this awesome update:

1. What bothers me the most is the chatgpt-shell-model-version variable and function. The variable can be either an index or a string, which chatgpt-shell.el uses to indicate the model. However, in this new version, the models are now structured as '(repeat (alist :key-type symbol :value-type sexp)), and keeping chatgpt-shell-model-version as a string seems a bit unnecessary. It also looks like it would require a lot of work for compatibility.
2. I think renaming chatgpt-shell-model-versions to chatgpt-shell-models might make more sense. What do you think? We already have the actual version inside chatgpt-shell-openai-models -> :version.

I am going to submit a PR for suggestion No. 2 (shouldn't be hard to change), and I'll leave suggestion No. 1 here for discussion.

PR: #247

[xenodium]

@xenodium Amazing work!

Thank you! and thanks for taking it for a spin and providing feedback so quickly.

What bothers me the most is the chatgpt-shell-model-version variable and function.

Ooh, if this is the biggest complain, I'm happy ;) Multi model support needed chunky changes throughout. I was anticipating more issues. I'll keep my fingers crossed 🤞.

The variable can be either an index or a string, which chatgpt-shell.el uses to indicate the model. However, in this new version, the models are now structured as '(repeat (alist :key-type symbol :value-type sexp)), and keeping chatgpt-shell-model-version as a string seems a bit unnecessary. It also looks like it would require a lot of work for compatibility.

Keeping chatgpt-shell-model-version should enable a smoother transition to the relatively higher number users setting this value in their configs. If they happen to set to a known version, everything should continue working. Otherwise, they should hopefully get a descriptive error message.

I think renaming chatgpt-shell-model-versions to chatgpt-shell-models might make more sense. What do you think?

Renaming SGTM. We'll also need a little migration logic to ask users to stop setting this variable (they are typically far fewer than those setting chatgpt-shell-model-version). I can follow-up with that.

I am going to submit a PR for suggestion No. 2 (shouldn't be hard to change), and I'll leave suggestion No. 1 here for discussion.

Thanks! I'll go take a look at the PR.

[xenodium]

Thanks! I'll go take a look at the PR.

Merged!

[xenodium]

We now have initial implementations for Claude, Gemini, and Ollama. Gonna close this feature request as all the majority of the work to go multi-model is now completed and the mentioned models working.

For anyone who had been anticipating multi-model support, please consider sponsoring. There was quite a bit of work needed to get here.