Support for HuggingFace models?

[gubowen2]

Hello, I think this is a great project and I would love to see it progress. Just wondering if there are plans to add support for HF models so that the optimization can happen locally. As a worker for hospitals, most data I am work with is PHI data, which is not allowed to leave the internet network. So the support for HF models will be greatly appreciated

[mertyg]

Hey @gubowen2 — thank you for the kind words! Yes for sure, making things work with HF / Open Source models is a big priority for us!

[saisurajkarra]

I did try to create my own HF engine and try to make gradient on the response but it fails

[HelloWorldLTY]

Thanks, it wll be a great enhancement.

[LengyanReader]

can I ask how long the support for opensource models can come up?

[vinid]

Hello! Support for open-source is already here! we have quite a few engines that cover most of open-source models. We don't yet have a hugging-face-specific implementation. See also this implementation