outlines icon indicating copy to clipboard operation
outlines copied to clipboard

Published 20 hours ago • verified publisher icon dottxt-ai

Groq Integration

Open rv4102 opened this issue 1 year ago • 4 comments

Since Groq follows OpenAI API, would it be possible to add support for it in the library?

rv4102 avatar Jun 08 '24 13:06 rv4102

As far as I understand, this might not be possible as Groq OpenAI API interface doesn't support logprobs:

Groq Cloud Documentation (https://console.groq.com/docs/openai): image

@rlouf am I correct in my understanding?

sharanry avatar Jun 23 '24 07:06 sharanry

Has anybody been able to run this for Groq?

andupotorac avatar Aug 12 '24 09:08 andupotorac

@andupotorac @sharanry I don't see any documentation supporting structured generation.

Even if a logit_bias argument was available, we still wouldn't be able to perform structured generation. In structured generation, the token filter changes every generation step.

lapp0 avatar Aug 12 '24 18:08 lapp0

Might be helpful to try to reach out to them and see if they implement something like you need?

andupotorac avatar Aug 12 '24 21:08 andupotorac

I wonder if SambdaNova supports them? https://community.sambanova.ai/t/sambanova-cloud-api-reference/197

metaskills avatar Dec 21 '24 18:12 metaskills