Portkey-AI
Translation of Stop Reason for Anthropic -> OpenAI
https://github.com/Portkey-AI/gateway/blob/fe0899bbf1a0f65f8f751aca8fca1b0a4409acb4/src/providers/anthropic/chatComplete.ts#L397
OpenAI SDKs with strict response validation (such as https://docs.rs/async-openai/latest/async_openai/) fails due to lack of mapping of Anthropic stop reason to a valid OpenAI stop reason.
We have PR's pending review to fix this. Will get them merged after necessary changes
Hey @functorism, @narengogi & team have decided to not take this up. The reason is explained in the linked PR. What are your thoughts?
Well, we're currently running a patch/fork of https://docs.rs/async-openai/latest/async_openai/ to get around this issue; so can't say I think it's the right move - but understand.
One way to view it is that Portkey is not an OpenAI compatible gateway unless things like these behave in expected ways.
Thorny issue. We haven't updated our thinking here yet, even though I totally understand that it makes the Gateway behave in unexpected ways
Would it be a better idea to have our own Rust SDK at one point? We ran into the same issue with OpenAI's official C# library recently
Right now, I'm assuming that OpenAIs Api is the standard to which Portkey is trying to make all other LLM APIs conform. Is that the goal Portkey has?
Something needs to be done here to try to convert. https://github.com/Portkey-AI/gateway/blob/ff4bfb965721542521916e3487f3301f81deecda/src/providers/anthropic/complete.ts#L94
It's a good thing it's centralized to be easy to fix, though.
Here is the solution for this particular one. https://github.com/braintrustdata/braintrust-proxy/blob/82c1a6732fe31db44417fdccddd23f1c9d7e494a/packages/proxy/src/providers/anthropic.ts#L284-L285
If you're facing issues with supporting an official OpenAI sdk - doesn't that tell you concretely that you're not meeting expected compliance?
In my mind the answer is pretty straight forward - maintain best effort mappings that ensures API spec compliance.
I also don't see an SDK as a solution - if the barrier to using portkey is using a portkey SDK, then that negates the point of portkey being an API level proxy - it rather becomes a library instead - and would compete with other popular solutions like LangChain.
This is very helpful @functorism @note89, thank you. Tagging @roh26it & @ayush-portkey again for visibility, and we'll get back to you with more thoughts on this
We do have a tag for strictOpenAiCompliance
Maybe we should adhere to OpenAI unless that check is turned off. (By default off in our SDKs)
The dilemma we're facing is that other APIs have diverged more and more from the OpenAI spec, and we have to make a hard call on how to support all the various APIs without compromising on features anymore. Prompt caching is one of the examples here.
