petals
petals copied to clipboard
bigscience-workshop
IndexError: tuple index out of range
To get bootstrapped, I tried to use the example from Readme
from transformers import AutoTokenizer
from petals import AutoDistributedModelForCausalLM
import torch
# Choose any model available at https://health.petals.dev
model_name = "petals-team/StableBeluga2" # This one is fine-tuned Llama 2 (70B)
# Connect to a distributed network hosting model layers
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoDistributedModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float32)
# Run the model as if it were on your computer
inputs = tokenizer("A cat sat", return_tensors="pt")["input_ids"]
outputs = model.generate(input_ids=inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0])) # A cat sat on a mat...
The torch_dtype=torch.float32 was added due to CPU support warning but apart from that rest is the same as the original example yet I am facing the error and unable to complete the inference.
OS : Ubuntu 22.04 CPU : i7-7700K GPU: Nvidia 1070
Please guide if i am missing something here.
+1
running into the same error while trying to generate by calling model.generate() method in the getting started colab notebook.
this seems to be an issue with the petals library itself instead of the transformers library since replacing AutoDistributedModelForCausalLM with AutoModelForCausalLM seems to work fine
@daspartho, I have the same thoughts as I have been using the same models for months directly, and it works. I just wanted to validate that it is not something due to my misconfiguration or overlooking some setting; we need to isolate the culprit and work towards PR since other folks are also facing this.
