text-generation-inference icon indicating copy to clipboard operation
text-generation-inference copied to clipboard
verified publisher icon huggingface

TGI with c4ai GPTQ models

Open vitalyshalumov opened this issue 1 year ago • 2 comments

System Info

tgi latest docker

Information

  • [X] Docker
  • [ ] The CLI directly

Tasks

  • [X] An officially supported command
  • [ ] My own modifications

Reproduction

trying to serve: alpindale/c4ai-command-r-plus-GPTQ results in : AssertionError: Tensor-likes are not equal! Missmatched elements:12160 / 12288

NEURALDEEPTECH/command-r-gptq or Cyleux/command-r-gptq results in: "AssertionError ("First input (bf16) and second input (fp32) must have the same dtypes!"

Expected behavior

served models

vitalyshalumov avatar Apr 14 '24 11:04 vitalyshalumov

Can you show the command you're using ?

Also show all the logs here please, we can't help without actual information and reproducibility.

Narsil avatar Apr 15 '24 07:04 Narsil

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] avatar May 16 '24 01:05 github-actions[bot]