text-generation-webui
text-generation-webui copied to clipboard
oobabooga
Character duplication at the start of each generation
Describe the bug
The last letter in the input box always gets duplicated in the generation output. See the screenshots.
Is there an existing issue for this?
- [X] I have searched the existing issues
Reproduction
Command line: python server.py --gptq-bits 4 --model llama-30b --gptq-pre-layer 7 Press generate, observe the text output
Screenshot
Logs
No error message, the lastest log messages are
(textgen) fgdfgfthgr@fgdfgfthgr-MS-7C95:/mnt/7018F20D48B6C548/text-generation-webui$ python server.py --gptq-bits 4 --model llama-30b --gptq-pre-layer 6
Loading llama-30b...
Loading model ...
Done.
Loaded the model in 38.19 seconds.
/home/fgdfgfthgr/anaconda3/envs/textgen/lib/python3.10/site-packages/gradio/deprecation.py:40: UserWarning: The 'type' parameter has been deprecated. Use the Number component instead.
warnings.warn(value)
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
System Info
OS: Linux Mint 21.1
GPU: 1050 ti
CPU: Ryzen 5500
Model: Llama 30b in 4 bit (but can replicate in Llama 7b)
Environment: Pycharm, anaconda:
# packages in environment at /home/fgdfgfthgr/anaconda3/envs/textgen:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
accelerate 0.17.1 pypi_0 pypi
aiofiles 23.1.0 pypi_0 pypi
aiohttp 3.8.4 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
altair 4.2.2 pypi_0 pypi
anyio 3.6.2 pypi_0 pypi
async-timeout 4.0.2 pypi_0 pypi
attrs 22.2.0 pypi_0 pypi
bitsandbytes 0.37.1 pypi_0 pypi
bzip2 1.0.8 h7b6447c_0
ca-certificates 2022.12.7 ha878542_0 conda-forge
certifi 2022.12.7 pyhd8ed1ab_0 conda-forge
charset-normalizer 3.1.0 pypi_0 pypi
click 8.1.3 pypi_0 pypi
cmake 3.26.0 pypi_0 pypi
contourpy 1.0.7 pypi_0 pypi
cudatoolkit-dev 11.7.0 h1fa729e_5 conda-forge
cycler 0.11.0 pypi_0 pypi
datasets 2.10.1 pypi_0 pypi
dill 0.3.6 pypi_0 pypi
entrypoints 0.4 pypi_0 pypi
fastapi 0.95.0 pypi_0 pypi
ffmpy 0.3.0 pypi_0 pypi
filelock 3.10.0 pypi_0 pypi
flexgen 0.1.7 pypi_0 pypi
fonttools 4.39.2 pypi_0 pypi
frozenlist 1.3.3 pypi_0 pypi
fsspec 2023.3.0 pypi_0 pypi
gradio 3.18.0 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
httpcore 0.16.3 pypi_0 pypi
httpx 0.23.3 pypi_0 pypi
huggingface-hub 0.13.2 pypi_0 pypi
idna 3.4 pypi_0 pypi
jinja2 3.1.2 pypi_0 pypi
jsonschema 4.17.3 pypi_0 pypi
kiwisolver 1.4.4 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1
libffi 3.4.2 h6a678d5_6
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
libuuid 1.41.5 h5eee18b_0
linkify-it-py 2.0.0 pypi_0 pypi
lit 15.0.7 pypi_0 pypi
markdown 3.4.1 pypi_0 pypi
markdown-it-py 2.2.0 pypi_0 pypi
markupsafe 2.1.2 pypi_0 pypi
matplotlib 3.7.1 pypi_0 pypi
mdit-py-plugins 0.3.3 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
multidict 6.0.4 pypi_0 pypi
multiprocess 0.70.14 pypi_0 pypi
ncurses 6.4 h6a678d5_0
networkx 3.0 pypi_0 pypi
numpy 1.24.2 pypi_0 pypi
nvidia-cublas-cu11 11.10.3.66 pypi_0 pypi
nvidia-cuda-cupti-cu11 11.7.101 pypi_0 pypi
nvidia-cuda-nvrtc-cu11 11.7.99 pypi_0 pypi
nvidia-cuda-runtime-cu11 11.7.99 pypi_0 pypi
nvidia-cudnn-cu11 8.5.0.96 pypi_0 pypi
nvidia-cufft-cu11 10.9.0.58 pypi_0 pypi
nvidia-curand-cu11 10.2.10.91 pypi_0 pypi
nvidia-cusolver-cu11 11.4.0.1 pypi_0 pypi
nvidia-cusparse-cu11 11.7.4.91 pypi_0 pypi
nvidia-nccl-cu11 2.14.3 pypi_0 pypi
nvidia-nvtx-cu11 11.7.91 pypi_0 pypi
openssl 1.1.1t h7f8727e_0
orjson 3.8.7 pypi_0 pypi
packaging 23.0 pypi_0 pypi
pandas 1.5.3 pypi_0 pypi
peft 0.2.0 pypi_0 pypi
pillow 9.4.0 pypi_0 pypi
pip 23.0.1 py310h06a4308_0
psutil 5.9.4 pypi_0 pypi
pulp 2.7.0 pypi_0 pypi
pyarrow 11.0.0 pypi_0 pypi
pycryptodome 3.17 pypi_0 pypi
pydantic 1.10.6 pypi_0 pypi
pydub 0.25.1 pypi_0 pypi
pyparsing 3.0.9 pypi_0 pypi
pyrsistent 0.19.3 pypi_0 pypi
python 3.10.9 h7a1cb2a_2
python-dateutil 2.8.2 pypi_0 pypi
python-multipart 0.0.6 pypi_0 pypi
pytz 2022.7.1 pypi_0 pypi
pyyaml 6.0 pypi_0 pypi
quant-cuda 0.0.0 pypi_0 pypi
readline 8.2 h5eee18b_0
regex 2022.10.31 pypi_0 pypi
requests 2.28.2 pypi_0 pypi
responses 0.18.0 pypi_0 pypi
rfc3986 1.5.0 pypi_0 pypi
safetensors 0.3.0 pypi_0 pypi
sentencepiece 0.1.97 pypi_0 pypi
setuptools 65.6.3 py310h06a4308_0
six 1.16.0 pypi_0 pypi
sniffio 1.3.0 pypi_0 pypi
sqlite 3.41.1 h5eee18b_0
starlette 0.26.1 pypi_0 pypi
sympy 1.11.1 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
tokenizers 0.13.2 pypi_0 pypi
toolz 0.12.0 pypi_0 pypi
torch 2.0.0 pypi_0 pypi
tqdm 4.65.0 pypi_0 pypi
transformers 4.28.0.dev0 pypi_0 pypi
triton 2.0.0 pypi_0 pypi
typing-extensions 4.5.0 pypi_0 pypi
tzdata 2022g h04d1e81_0
uc-micro-py 1.0.1 pypi_0 pypi
urllib3 1.26.15 pypi_0 pypi
uvicorn 0.21.1 pypi_0 pypi
websockets 10.4 pypi_0 pypi
wheel 0.38.4 py310h06a4308_0
xxhash 3.2.0 pypi_0 pypi
xz 5.2.10 h5eee18b_1
yarl 1.8.2 pypi_0 pypi
zlib 1.2.13 h5eee18b_0
(textgen) fgdfgfthgr@fgdfgfthgr-MS-7C95:/mnt/7018F20D48B6C548/text-generation-webui$ pypi_0 pypi
websockets 10.4 pypi_0 pypi
wheel 0.38.4 py310h06a4308_0
xxhash 3.2.0 pypi_0 pypi
xz 5.2.10 h5eee18b_1
yarl 1.8.2 pypi_0
I've noticed the same thing. I'm also using LLaMA 7b 4bit, Ubuntu in WSL on windows 10.
Same here. I think it started like two or three days ago (around 2023-3-19)
The issue seems to be resolved in the last pull, at least I am not experiencing it anymore. Likely from commit #de6a09d