Support for OSS Models behind an API?

[rafaelvp-db]

🚀 The feature

Right now mainly proprietary LLMs are supported. Would be great to also support DIY/OSS LLMs - for instance, hosted in Databricks Model Serving endpoints. Or more holistically, LLMs deployed behind a Web API running in a container, for instance.

Motivation, pitch

I think this will be super useful for people or companies who are willing to do prompt engineering with Open Source LLMs. Also happy to work on this feature.

Alternatives

Could also allow for testing prompts targetting models that are running on the local machine

Additional context

No response

[NivekT]

Hi @rafaelvp-db,

Thanks for opening this issue! That sounds like a great integration for us to add! If you or anyone else would like to open a PR, we would be more than happy to review it or support you in any way.

We are also happy to discuss other potential integration with Databricks on our Discord or through other channels. Just let us know!