Added Support for the new /fim/completions route on Mistral AI API

[keshavkrishna]

Support new /fim/completions route on Mistral AI API:

Description:

• /fim/completions endpoint work in the same way as the /chat/completions route. It handles extra parameters such as suffix, prompt also.
• Implemented fimCompletionsHandler to handle requests to the /fim/completions endpoint.
• Updated the API configuration to include the new fimComplete endpoint.
• implemented response and stream chunk transformations for the fimComplete function in Mistral AI.
• Modified index.ts to register the new route v1//fim/completions with request validation.

Motivation:

• To add support in portkey gateway for the new /fim/completions route on Mistral AI API

Related Issues:

• #381

[narengogi]

Thank you for the contribution @keshavkrishna will review this soon

[VisargD]

Hey @keshavkrishna! Will you be able to take a look at the comments? And please update your branch with the latest main branch changes

[keshavkrishna]

Yes I will take care of these comments by this weekend. Is that timeline okay?

[VisargD]

Hey. Is it not possible to reuse the existing /chat/completions route? We can passthrough all the extra key as it is

[keshavkrishna]

Requests to the existing /chat/completions route are redirected to the provider's base URL with the endpoint chat/completions. In this scenario, we need to direct requests to the provider's base URL with the endpoint fim/completions.

[VisargD]

We can accept a new header x-portkey-mistral-fim-completion: true and route it conditionally to the the mistral fim completions route. Adding a new router for provider specific routes is not the most feasible solution here because it will increase the number of routers in future.

EDIT: Adding extra details

• In this function, gateway already passes providerOptions. Using it, you can check for a new param called providerOptions.mistralFimCompletion. If its true, you can change the endpoint to the mistral fim endpoint. https://github.com/Portkey-AI/gateway/blob/9724792edf7ff7954d068f00b25f90b5009a3cd0/src/providers/mistral-ai/api.ts#L8
• In the chatComplete config, you can add the extra fields which are required (like suffix, prompt, etc.). The response structure seems to be OpenAI compliant. So that should not be a problem

[keshavkrishna]

Sure, I'll implement this

[keshavkrishna]

@VisargD

If I am not getting mistralFimCompletion in providerOptions. following is provider option i am getting

{
  provider: 'mistral-ai',
  apiKey: '  SOME_API_KEY_HERE',
  overrideParams: {},
  retry: {},
  cache: {}
} 

so i have passed requestHeaders in getEndpioint, and i am checking by if (requestHeaders['x-portkey-mistral-fim-completion'] === 'true')

Is that okay??

[VisargD]

@VisargD

If I am not getting mistralFimCompletion in providerOptions. following is provider option i am getting

{
  provider: 'mistral-ai',
  apiKey: '  SOME_API_KEY_HERE',
  overrideParams: {},
  retry: {},
  cache: {}
} 

so i have passed requestHeaders in getEndpioint, and i am checking by if (requestHeaders['x-portkey-mistral-fim-completion'] === 'true')

Is that okay??

Hey! specific headers checks might not be required. Here is what you can do:

• In this function, add one more condition for mistral-ai provider and add the mapping for mistralFimCompletion in it. Check the existing conditions for other providers for reference. Once this is done, you will always get the mistralFimCompletion param in providerOptions. https://github.com/Portkey-AI/gateway/blob/3bb3a80385258ab2b2262ac28f6e3f2f47e0e3f1/src/handlers/handlerUtils.ts#L969

Please let me know if you want an example

[keshavkrishna]

Yes it is. We are using chatComplete transforms as per visarg's comment.

[VisargD]

@keshavkrishna - Can you please run npm run format and also update with main branch? I can merge it now

[keshavkrishna]

@VisargD as requested i have updated my branch and also formatted it