Add new Gradio web demo for Chinese-LLaMA-Alpaca

[sunyuhan19981208]

I write a simple web gradio demo which supports multi-round conversation and multi-card inference. Here is the shell command to start the web demo:

python gradio_demo.py --base_model /home/sunyuhan/syh/sunyuhan/zju/llama-7b-hf/ --lora_model /home/sunyuhan/syh/sunyuhan/zju/chinese-alpaca-lora-7b --with_prompt --gpus 4,5,6,7

[airaria]

Looks awesome! Thank you for contribution. But I have a question about loading Alpaca Plus models. Since the Plus needs two LoRA weights, so does the demo support multi-LoRA loading? Or alternatively, does the it support loading only base_model without lora_model (since users can merge the LoRAs into the base model to get a single merged Plus model weight file)?

[sunyuhan19981208]

support base_model-only mode

[airaria]

support base_model-only mode

Thanks. We will perform some tests before confirming merging.

[sunyuhan19981208]

support base_model-only mode

Thanks. We will perform some tests before confirming merging.

Please let me know if you have any advice or if you've found any bugs as soon as possible. I would be grateful to receive your feedback, and I'm eager to work on improving based on your suggestions. Thank you very much.

[tluo-github]

在这个 notebook 中: https://github.com/ymcui/Chinese-LLaMA-Alpaca/blob/main/notebooks/convert_and_quantize_chinese_llama.ipynbusp=sharing#scrollTo=EjkXqaqbmrVZ 使用如下命令: 能成功运行 !cd Chinese-LLaMA-Alpaca/ && python scripts/my_gradio_demo.py \ --base_model 'decapoda-research/llama-7b-hf' \ --lora_model 'ziqingyang/chinese-alpaca-lora-7b'

在这个 notebook ： https://github.com/ymcui/Chinese-LLaMA-Alpaca/blob/main/notebooks/finetune_chinese_alpaca_lora.ipynb !cd Chinese-LLaMA-Alpaca/ && python scripts/my_gradio_demo.py \ --base_model 'decapoda-research/llama-7b-hf' \ --lora_model '/content/output_model/peft_model' 运行起来之后，提问会报错: ` 2023-05-15 08:58:49.995893: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT Loading checkpoint shards: 100% 33/33 [00:18<00:00, 1.75it/s] Vocab of the base model: 32000 Vocab of the tokenizer: 49954 Resize model embeddings to fit tokenizer loading peft model Running on local URL: http://127.0.0.1:7860/ Running on public URL: https://da6b02733e495413ee.gradio.live/

This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 414, in run_predict output = await app.get_blocks().process_api( File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1320, in process_api result = await self.call_function( File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1048, in call_function prediction = await anyio.to_thread.run_sync( File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, *args) File "/content/Chinese-LLaMA-Alpaca/scripts/my_gradio_demo.py", line 132, in predict generation_output = model.generate( File "/usr/local/lib/python3.10/dist-packages/peft/peft_model.py", line 581, in generate outputs = self.base_model.generate(**kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 1524, in generate return self.beam_search( File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2810, in beam_search outputs = self( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py", line 687, in forward outputs = self.model( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py", line 577, in forward layer_outputs = decoder_layer( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py", line 292, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py", line 196, in forward query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/peft/tuners/lora.py", line 358, in forward result += self.lora_B(self.lora_A(self.lora_dropout(x))) * self.scaling File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/linear.py", line 114, in forward return F.linear(input, self.weight, self.bias) RuntimeError: expected scalar type Half but found Float Keyboard interruption in main thread... closing server.`

[tluo-github]

已解决: load_type = torch.float16 改为如下: load_type = torch.float32