Allow alternative OpenAI-compatible endpoints

[betschki]

Context

This PR introduces functionality to allow users to specify a custom OpenAI-compatible API endpoint in addition to the default OpenAI API. The main goal behind this is to enable the usage of alternative endpoints such as self-hosted or third-party services. At stadt.werk, we use this feature to interact with our own self-hosted OpenAI-compatible server.

The enhancement extends the flexibility of the existing interface by letting users switch between OpenAI’s official endpoint (default behaviour) and their own custom endpoint through the UI. Additionally, the PR refines model fetching, which is essential for any non-OpenAI endpoints.

Compatability

• This change is fully backward-compatible with existing OpenAI API usage. If no custom endpoint is specified, the application will default to using OpenAI’s API as before.
• The enhancements have been tested with both OpenAI’s API and OpenAI-compatible llama.cpp infrastructure.
• Users who do not require a custom endpoint will not be affected by these changes.

We'd love to see this enhancement merged into chatgpt-web, since we believe this can be a great feature that has also been mentioned a few times (here and here, for example).

Looking forward to feedback.

[Niek]

Great work! I have a few questions:

• How does this play together with VITE_API_BASE? I see you still support it, I suppose it's good to keep that in as a way to default to some other value than OpenAI?
• I tried with the Anthropic API, but I get CORS errors. We might need to add some exceptions for specific popular APIs
• I think it makes sense to "test" a newly entered API endpoint by calling /models before storing it
• Maybe it makes sense to remove Petals support and recommend https://docs.litellm.ai/docs/simple_proxy instead

[betschki]

Hey @Niek!

Thank you for your feedback, appreciate it.

To answer your questions:

• Yes, we did not want to impact current functionality, so if no custom endpoint is set, VITE_API_BASE will still work. That was important from our end to ensure nothing breaks during a potential update, in case somebody set the variable and expects it to keep working.
• We have run into the same issue on our llamacpp server. From a client perspective, there is not too much we can do in an agnostic way, in my eyes. Anthropic has added CORS suppport as of recently, though this requires setting an additional header. Not quite sure how this can be handled effectively. Hard-coding configurations for popular APIs is an option, though this can quickly become a maintenance nightmare (and would still not guarantee that all APIs will work browser-based).
• Great idea with sending a request to /models before storing an endpoint. Will implement that today and update th PR.
• We haven't touched Petals, since none of us uses it or has experience with it. Always a fan of simplifying though - and this could be a great opportunity for that, indeed. I would see that out of scope for this specific PR though.

So, I'll implement that call to /models and will wait for your feedback on CORS.

[betschki]

@Niek I have added the check for the /models endpoint before saving the endpoint in the settings.

Looking forward to your feedback regarding CORS.

[Niek]

Thanks a lot, merged now @betschki!

I tried to get it working with Anthropic, but no luck. The changes needed are:

• Authentication is done with x-api-key instead of the Authorization header
• anthropic-dangerous-direct-browser-access: true needs to be set on all API calls (small change)
• the /models endpoint is non-existent, quite ridiculous if you ask me (see: https://github.com/anthropics/anthropic-sdk-typescript/issues/458)

So for now, this is only available for 100% OpenAI-compatible endpoints like llama.cpp.

[Niek]

@all-contributors please add @betschki for code, ideas

[allcontributors[bot]]

@Niek

@betschki already contributed before to code, ideas

[akarelas]

I tried to get it working with Anthropic, but no luck.

Sounds like an important enough change to implement though, because Anthropic is better in many use-cases, like it's better for tech & coding than OpenAI.

[Niek]

I tried to get it working with Anthropic, but no luck.

Sounds like an important enough change to implement though, because Anthropic is better in many use-cases, like it's better for tech & coding than OpenAI.

Agreed, it would be nice if the code could be restructured a bit so all API requests are done through requests.svelte. There we can add some logic depending on the hostname.