Track PromptNode tested HF local LLMs

[vblagoje]

We should compile a list of verified LLM models to manage the ever-expanding selection of open-source LLMs from HuggingFace (HF), available on the HF Hub. Accompanying each model, we should provide a related Colab notebook. This approach will allow our users to grasp the models we tried with PromptNode quickly and to start using them promptly (pun intended).

Here are the models we have tried already and the accompanying colabs:

• OpenLLama - Colab
• Falcon - Colab
• Guanaco - Colab needed
• StarChat - Colab needed
• SantaCoder - Colab needed
• Mosaic Instruct - Colab
• Mosaic Chat - Colab
• Baichuan - Colab needed

[anakin87]

(I still haven't experimented with these models.)

Another facet of this exploration work might involve loading models using quantization (Hugging Face blog post) and see if it requires some dependencies.

[anakin87]

Haystack does not support quantization out-of-the-box, but does not seem to have any problems using Transformer models loaded with the accelerate and bitsandbytes libraries. Our docs

• LLama v2 - Colab

[Timoeller]

Seems out of date, closing for now.