Prevent first part of KoboldAI API responses from getting cut off with --chat enabled

[AICatgirls]

This is related to issue 1053

I'm not sure what the intent of the [len(prompt):] part was, but it causes the first part of API responses to be cut off.

[oobabooga]

This API was created to mimic the kobold ai interface such that the web UI can be used as a backend to https://github.com/TavernAI/TavernAI. It seems like it is necessary to remove the original prompt from the response and only return the completion. Can you verify?

[AICatgirls]

I tested both TavernAI and SillyTavern using the API:KoboldAI and GUI KoboldAI Settings. Neither worked before my change, and both worked afterwards. It looks like the [len(prompt):] part was added 4 days ago, but that's as far as I've researched so I'm not sure why.

[oobabooga]

The original PR for this extension is this one:

https://github.com/oobabooga/text-generation-webui/pull/342

Line 67 includes answer[len(prompt):]:

https://github.com/oobabooga/text-generation-webui/pull/342/files#diff-e2c5630e43cf5c117d8e71cd7d09ae763aafb80c7210dcd8b90cdc7c381d5aceR67

I'm confused about this.

[AICatgirls]

Could the change have been in modules/text_generation.py in the generate_reply function? I notice that truncation_length was recently added. Maybe the the substring is now being taken further upstream?

In case it makes a difference for anyone reproducing my testing, my start-webui.bat has call python server.py --auto-devices --chat --wbits 4 --groupsize 128 --listen --model gpt4-x-alpaca-13b-native-4bit-128g --pre_layer 28 --extensions api in it.

[oobabooga]

Maybe @mayaeary can shed some light

[janvarev]

@AICatgirls I had the same problem - you just need to run bat file WITHOUT --chat flag, and everything works.

[AICatgirls]

@AICatgirls I had the same problem - you just need to run bat file WITHOUT --chat flag, and everything works.

Thank you, that explains it. I've modified script.py to check the --chat flag, and if it is enabled then it will give the straight answer response, otherwise it will give the substring.

[Ph0rk0z]

All the tavern stuff said to run in notebook mode.

[AICatgirls]

Tested in --notebook mode, and it works. Also tested --cai-chat and it works as well now because --cai-chat sets shared.args.chat=true.

[oobabooga]

Thanks, @AICatgirls!

[AICatgirls]

@oobabooga I'm getting an error now

Traceback (most recent call last): File "D:\oobabooga\installer_files\env\lib\socketserver.py", line 683, in process_request_thread self.finish_request(request, client_address) File "D:\oobabooga\installer_files\env\lib\socketserver.py", line 360, in finish_request self.RequestHandlerClass(request, client_address, self) File "D:\oobabooga\installer_files\env\lib\socketserver.py", line 747, in init self.handle() File "D:\oobabooga\installer_files\env\lib\http\server.py", line 433, in handle self.handle_one_request() File "D:\oobabooga\installer_files\env\lib\http\server.py", line 421, in handle_one_request method() File "D:\oobabooga\text-generation-webui\extensions\api\script.py", line 76, in do_POST 'text': answer if shared.args.is_chat() else answer[len(prompt):] AttributeError: 'Namespace' object has no attribute 'is_chat'

I think you might have intended: 'text': answer if shared.is_chat() else answer[len(prompt):]

[appleguy]

Yes, same thing here on a brand new install, so this is likely to break others.

@AICatgirls This change causes Tavern.AI to fail when connected to the API, with this Python exception:

Traceback (most recent call last):
  File "/home/tech/AI/oobabooga_linux/installer_files/env/lib/python3.10/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "/home/tech/AI/oobabooga_linux/installer_files/env/lib/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/tech/AI/oobabooga_linux/installer_files/env/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()
  File "/home/tech/AI/oobabooga_linux/installer_files/env/lib/python3.10/http/server.py", line 433, in handle
    self.handle_one_request()
  File "/home/tech/AI/oobabooga_linux/installer_files/env/lib/python3.10/http/server.py", line 421, in handle_one_request
    method()
  File "/home/tech/AI/oobabooga_linux/text-generation-webui/extensions/api/script.py", line 76, in do_POST
    'text': answer if shared.args.is_chat() else answer[len(prompt):]
AttributeError: 'Namespace' object has no attribute 'is_chat'

[oobabooga]

Fixed here https://github.com/oobabooga/text-generation-webui/commit/7ff645899e4610b16574bdd22a4d154c93d5b830

[Nexus80]

I get exactly the same error. When I loaded up the --extensions api from text-generation-webui I could first get connected in TavernAI, when I choose a character I immediately lost the connection in TavernAI. So something is really strange or bugged in here.

I would really appreciate a fix for this thing :)