Sema4AI
Action Server does not handle output size
What happened?
- Current Status *
Current Action Server @action has limitless return value size. OpenAI GPT and Large Language Models in general work with limited size tokenized texts. OpenAI GPT allowed return value size is around 3500 ASCII characters. After that it will just error on Action use.
@action
def kaboom():
return "A"*10000
This means that actions that return too big outputs will always fail from the callers (clients) point of view.
- How should it work *
I propose that Action Server internally truncates the outputs. The exact max size number could be configured in the Web UI.
Reference implementation:
def truncate_output_with_beginning_clue(output: str, max_chars: int = 2000) -> str:
beginning_clue = "[Cut] " # A very short clue at the beginning to indicate possible truncation
if len(output) > max_chars:
truncated_output = output[:max_chars - len(beginning_clue)]
chars_missed = len(output) - len(truncated_output)
truncated_message = f"[+{chars_missed}]"
return beginning_clue + truncated_output + truncated_message
else:
return output
System Info
- 0.0.13 * Python 3.11.5 * OSX *
OpenAI GPT allowed return value size is around 3500 ASCII characters
Doesn't look like GPTs would restrict this anymore (at least there is not an error), but the GPT will not have access to the entire output if it's long. When asking a GPT what it can handle, the response is:
The maximum length of text I can effectively handle in a single interaction depends on the context of the request. For processing and analyzing text within a single response, I work best with segments up to around 4096 tokens (words and punctuation marks). This limit helps ensure accuracy in my responses and analyses.
Note: Action Server Toolkit for langchain has a truncation in the connector. Here.
Note: Action Server Toolkit for langchain has a truncation in the connector. => this is not the case anymore.