distilabel
distilabel copied to clipboard
argilla-io
[FEATURE] Thinking of adding a GroqLLM integration to distilabel
Groq is a startup developing LPU engines which provide 18x faster inference than GPUs. They currently provide Llama2-70B via API, and Mixtral 8x7B access on request. It would be great to add a GroqLLM module to distilabel just like we have OpenAILLM. I've been trying out their platform and it's 🔥 .
It shouldn't be too difficult because they already try to make switching from OpenAI to Groq API easy.
Yes, Groq is lightning fast and we're aware of it! We're now going through a distilabel refactor, but for sure that's something we'll have into consideration and try to implement as part of the v1.0 later this month. Thanks for the proposal!
Nice! What does distilabel "refactor" mean though? Also I could try make a GroqLLM module in the meantime if that'd be good?
On Mon, Feb 19, 2024, 9:56 AM Alvaro Bartolome @.***> wrote:
Yes, Groq is lightning fast and we're aware of it! We're now going through a distilabel refactor, but for sure that's something we'll have into consideration and try to implement as part of the v1.0 later this month. Thanks for the proposal!
— Reply to this email directly, view it on GitHub https://github.com/argilla-io/distilabel/issues/348#issuecomment-1952760023, or unsubscribe https://github.com/notifications/unsubscribe-auth/A43V2KUTZOJYE6IJ4AAXBITYUNY4LAVCNFSM6AAAAABDPVXMOSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJSG43DAMBSGM . You are receiving this because you authored the thread.Message ID: @.***>
So we saw some limitations on certain scenarios and we're now trying to provide more flexibility, so we decided to refactor it and ship it as v1.0, that's why we won't be releasing minor releases until the refactor is done (expect it by the end of the month more or less). But sure, if you're willing to contribute feel free to include it, as LLM won't change much so would be appreciated and useful! Just let me know if you need any other detail 👍🏻
@kcentric adding GroqLLM now would be super cool indeed, that means we and the community can start testing it right away!
