How to use model without api with guard

[MuhammadHammadBashir]

I have models locally downloaded and using from ctransformers import AutoModelForCausalLM from ctransformers import AutoModelForCausalLM, AutoConfig, Config conf = AutoConfig(Config(temperature=0.8, repetition_penalty=1.1, batch_size=52, max_new_tokens=1024, context_length=2048)) llm = AutoModelForCausalLM.from_pretrained("/content/llama-2-13b-chat.Q4_K_S.gguf", model_type="llama", config = conf)

with api we can use like this res = guard( openai.chat.completions.create, prompt_params={"doctors_notes": doctors_notes}, max_tokens=1024, temperature=0.3, )

but how to use above model in this context

[zsimjee]

Hey Muhammad,

In that case, you would wrap the model invocation in a custom callable, following the docs here.

In this case it could be as simple as

def custom_llm_callable(prompt):
  return llm( #pass prompt to the llm here

res = guard(
  custom_llm_callable,
  prompt_params={"doctors_notes": doctors_notes},
  max_tokens=1024,
  temperature=0.3,
)

[CalebCourier]

@MuhammadHammadBashir we also recently added support for HuggingFace models. So if you are using one of those from the transformers library you can treat model.generate like an llm api.

We still need to add better docs for this functionality, but you can find a summary and example here: https://www.guardrailsai.com/blog/0.3-release/#anthropic-and-hugging-face-models-support

[MuhammadHammadBashir]

thank you for info.

On Fri, Jan 26, 2024 at 7:50 PM Caleb Courier @.***> wrote:

@MuhammadHammadBashir https://github.com/MuhammadHammadBashir we also recently added support for HuggingFace models. So if you are using one of those from the transformers library you can treat model.generate like an llm api.

We still need to add better docs for this functionality, but you can find a summary and example here:

https://www.guardrailsai.com/blog/0.3-release/#anthropic-and-hugging-face-models-support

— Reply to this email directly, view it on GitHub https://github.com/guardrails-ai/guardrails/issues/551#issuecomment-1912189183, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKJTVY6G5WUQFBRFQGGPSLTYQO7FHAVCNFSM6AAAAABCFTDTFSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJSGE4DSMJYGM . You are receiving this because you were mentioned.Message ID: @.***>

[dtam]

Hi @MuhammadHammadBashir as there's been no updates on this thread in a few months. If you have any questions or issues still please don't hesitate to reopen this or reach out to me.