eland icon indicating copy to clipboard operation
eland copied to clipboard

Support for TaylorAI/gte-tiny

Open Shifter2600 opened this issue 11 months ago • 0 comments

Receiving an error when loading model

Downloading: 100%|██████████████████████████████████████████████████████████████████████| 1.50k/1.50k [00:00<00:00, 980kB/s]
Downloading: 100%|███████████████████████████████████████████████████████████████████████| 226k/226k [00:00<00:00, 1.97MB/s]
Downloading: 100%|███████████████████████████████████████████████████████████████████████| 82.0/82.0 [00:00<00:00, 92.6kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████████████| 228/228 [00:00<00:00, 111kB/s]
Traceback (most recent call last):
  File "/usr/local/bin/eland_import_hub_model", line 197, in <module>
    tm = TransformerModel(args.hub_model_id, args.task_type, args.quantize)
  File "/usr/local/lib/python3.9/dist-packages/eland/ml/pytorch/transformers.py", line 567, in __init__
    self._tokenizer = transformers.AutoTokenizer.from_pretrained(
  File "/usr/local/lib/python3.9/dist-packages/transformers/models/auto/tokenization_auto.py", line 579, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/transformers/tokenization_utils_base.py", line 1783, in from_pretrained
    return cls._from_pretrained(
  File "/usr/local/lib/python3.9/dist-packages/transformers/tokenization_utils_base.py", line 1984, in _from_pretrained
    raise ValueError(
ValueError: Non-consecutive added token '[PAD]' found. Should have index 30522 but has index 0 in saved vocabulary.

Shifter2600 avatar Mar 23 '24 18:03 Shifter2600