FAQ Section

[psinger]

Would be great to have some FAQs and templates/notebooks for common questions.

• [ ] How to generate outputs outside of LLM Studio with trained weights pushed to HF
• [ ] How to continue training from previous experiments, how to load local weights
• [ ] (Lack-of) backward compatibility

[psinger]

For loading from HF

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("hf_path")
model = AutoModelForCausalLM.from_pretrained("hf_path")
model.half().cuda()

# need to match the input prompt how you are doing it in the LLM Studio Prompt
inputs = tokenizer("How are you?<|endoftext|>", return_tensors="pt").to("cuda")
tokens = model.generate(
  **inputs,
  max_new_tokens=64,
  temperature=0.7,
  do_sample=True,
)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))

[psinger]

Most has been already resolved.