[New Feature] [important] Adding LoRA based chatbot app (local web page)

[Dandelionym]

Hi, thanks for your @shizhediao open-source code sharing.

Is your feature request related to a problem? Please describe. Yes, there is currently no easy way to interact with LoRA-based chatbots through a GUI. Only command line interfaces are supported for fine-tuned LoRA models. An app with a web UI would make these chatbots much more accessible.

Describe the solution you'd like I would like to see the development of a chat app with a web UI that can interface with LoRA-based chatbots. The app should allow users to chat with different LoRA models through a simple web page, without needing to use the command line.

Describe alternatives you've considered The only alternative is to continue using the command line to chat with LoRA chatbots. A web app would open up these AI systems to many more casual users who prefer an easy-to-use interface.

Additional context LoRA or Learned Representations for Anthropic AI is a technique developed by Anthropic for aligning and constitutionalizing language models. LoRA allows researchers to build chatbots and other systems with natural language interfaces in a safe, robust, and beneficial manner. However, LoRA currently can only be accessed through the command line, limiting its reach. A web app chat interface for LoRA would help bring its benefits to more users.

[SHUMKASHUN]

Hi, Web UI now support Lora-based models, you can run scripts/run_app.sh to have a try by simply add --lora_model_path {YOUR_LORA_PATH} \ in the script.

[Dandelionym]

Hi, thank you, but s+till not yet. @SHUMKASHUN

Debugging middleware caught exception in streamed response at a point where response headers were already sent.
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/lmflow/lib/python3.9/site-packages/werkzeug/wsgi.py", line 500, in __next__
    return self._next()
  File "/home/ubuntu/anaconda3/envs/lmflow/lib/python3.9/site-packages/werkzeug/wrappers/response.py", line 50, in _iter_encoded
    for item in iterable:
  File "/home/ubuntu/anaconda3/envs/lmflow/lib/python3.9/site-packages/flask/helpers.py", line 149, in generator
    yield from gen
  File "/data/ExternalHome/lins_research/phd_projects/LMFlow-S/service/app.py", line 71, in stream_generate
    out = model.backend_model(torch.as_tensor([input_ids], device=local_rank), use_cache=True)
  File "/home/ubuntu/anaconda3/envs/lmflow/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/lmflow/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/lmflow/lib/python3.9/site-packages/peft/peft_model.py", line 575, in forward
    return self.base_model(
  File "/home/ubuntu/anaconda3/envs/lmflow/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/lmflow/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/lmflow/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 687, in forward
    outputs = self.model(
  File "/home/ubuntu/anaconda3/envs/lmflow/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/lmflow/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/lmflow/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 525, in forward
    position_ids = position_ids.unsqueeze(0).view(-1, seq_length)
RuntimeError: cannot reshape tensor of 0 elements into shape [-1, 0] because the unspecified dimension size -1 can be any value and is ambiguous

[SHUMKASHUN]

Hi, Could you share more information about this error (script command, your model and your input). It seems I have never meet this error before, so I may need more information to find out the issue.

[shizhediao]

This issue has been marked as stale because it has not had recent activity. If you think this still needs to be addressed please feel free to reopen this issue. Thanks