open-interpreter
open-interpreter copied to clipboard
OpenInterpreter
Open Files
Describe the bug
I tried to open a screenshot with this command and Open Interpreter Crashed:
`> open /Users/maxpetrusenko/Desktop/photo_2023-11-23_17-58-50.jpg please
Python Version: 3.10.12
Pip Version: 23.2.1
Open-interpreter Version: cmd:A, pkg: 0.1.15
OS Version and Architecture: macOS-14.1-arm64-arm-64bit
CPU Info: arm
RAM Info: 16.00 GB, used: 6.80, free: 0.19
Interpreter Info
Vision: False
Model: openai/gpt-4
Function calling: False
Context window: 3000
Max tokens: 1000
Auto run: False
API base: http://localhost:1234/v1
Local: True
Curl output: [Errno 2] No such file or directory: 'curl http://localhost:1234/v1'
Traceback (most recent call last): File "/Users/maxpetrusenko/miniforge3/lib/python3.10/site-packages/interpreter/core/respond.py", line 49, in respond for chunk in interpreter._llm(messages_for_llm): File "/Users/maxpetrusenko/miniforge3/lib/python3.10/site-packages/interpreter/llm/convert_to_coding_llm.py", line 65, in coding_llm for chunk in text_llm(messages): File "/Users/maxpetrusenko/miniforge3/lib/python3.10/site-packages/interpreter/llm/setup_text_llm.py", line 32, in base_llm messages = tt.trim( File "/Users/maxpetrusenko/miniforge3/lib/python3.10/site-packages/tokentrim/tokentrim.py", line 189, in trim shorten_message_to_fit_limit(message, tokens_remaining, model) File "/Users/maxpetrusenko/miniforge3/lib/python3.10/site-packages/tokentrim/tokentrim.py", line 95, in shorten_message_to_fit_limit new_length = int(len(encoding.encode(content)) * ratio) File "/Users/maxpetrusenko/miniforge3/lib/python3.10/site-packages/tiktoken/core.py", line 116, in encode if match := _special_token_regex(disallowed_special).search(text): TypeError: expected string or buffer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/maxpetrusenko/miniforge3/bin/interpreter", line 8, in
Please make sure LM Studio's local server is running by following the steps above.
If LM Studio's local server is running, please try a language model with a different architecture.`
Reproduce
- run interpreter --local
- start server ( LM Studio tried with mistral instruct v0 1 cguf )
- open "/path/to/screenshot" please
Expected behavior
no crash
Screenshots
No response
Open Interpreter version
0.1.15
Python version
3.11
Operating System name and version
mac m2
Additional context
No response
Seems like it cut of the image
Traceback (most recent call last): File "/Users/maxpetrusenko/miniforge3/lib/python3.10/site-packages/interpreter/core/respond.py", line 49, in respond for chunk in interpreter._llm(messages_for_llm): File "/Users/maxpetrusenko/miniforge3/lib/python3.10/site-packages/interpreter/llm/convert_to_coding_llm.py", line 65, in coding_llm for chunk in text_llm(messages): File "/Users/maxpetrusenko/miniforge3/lib/python3.10/site-packages/interpreter/llm/setup_text_llm.py", line 32, in base_llm messages = tt.trim( File "/Users/maxpetrusenko/miniforge3/lib/python3.10/site-packages/tokentrim/tokentrim.py", line 189, in trim shorten_message_to_fit_limit(message, tokens_remaining, model) File "/Users/maxpetrusenko/miniforge3/lib/python3.10/site-packages/tokentrim/tokentrim.py", line 95, in shorten_message_to_fit_limit new_length = int(len(encoding.encode(content)) * ratio) File "/Users/maxpetrusenko/miniforge3/lib/python3.10/site-packages/tiktoken/core.py", line 116, in encode if match := _special_token_regex(disallowed_special).search(text): TypeError: expected string or buffer
I got same error when I tried below: I'll try to fix it.
import interpreter
interpreter.auto_run = True
interpreter.vision = True
interpreter.chat("Please read text in this image /Users/user/Downloads/test.png")
-> % python test.py
Python Version: 3.10.12
Pip Version: 23.2.1
Open-interpreter Version: cmd:Interpreter, pkg: 0.1.16
OS Version and Architecture: macOS-14.1.1-arm64-arm-64bit
CPU Info: arm
RAM Info: 16.00 GB, used: 4.79, free: 0.20
Interpreter Info
Vision: True
Model: gpt-4
Function calling: True
Context window: None
Max tokens: None
Auto run: True
API base: None
Local: False
Curl output: Not local
Traceback (most recent call last):
File "/Volumes/T7/WorkSpace/personal/open-interpreter/interpreter/core/llm/setup_openai_coding_llm.py", line 59, in coding_llm
messages = tt.trim(
File "/opt/anaconda3/envs/open-interpreter/lib/python3.10/site-packages/tokentrim/tokentrim.py", line 189, in trim
shorten_message_to_fit_limit(message, tokens_remaining, model)
File "/opt/anaconda3/envs/open-interpreter/lib/python3.10/site-packages/tokentrim/tokentrim.py", line 95, in shorten_message_to_fit_limit
new_length = int(len(encoding.encode(content)) * ratio)
File "/opt/anaconda3/envs/open-interpreter/lib/python3.10/site-packages/tiktoken/core.py", line 116, in encode
if match := _special_token_regex(disallowed_special).search(text):
TypeError: expected string or buffer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Volumes/T7/WorkSpace/personal/open-interpreter/test.py", line 5, in <module>
interpreter.chat("この画像に書かれているテキストを教えてください。/Volumes/T7/WorkSpace/personal/open-interpreter/adjusted_thumbnail_image.png")
File "/Volumes/T7/WorkSpace/personal/open-interpreter/interpreter/core/core.py", line 80, in chat
for _ in self._streaming_chat(message=message, display=display):
File "/Volumes/T7/WorkSpace/personal/open-interpreter/interpreter/core/core.py", line 95, in _streaming_chat
yield from terminal_interface(self, message)
File "/Volumes/T7/WorkSpace/personal/open-interpreter/interpreter/terminal_interface/terminal_interface.py", line 121, in terminal_interface
for chunk in interpreter.chat(message, display=False, stream=True):
File "/Volumes/T7/WorkSpace/personal/open-interpreter/interpreter/core/core.py", line 116, in _streaming_chat
yield from self._respond()
File "/Volumes/T7/WorkSpace/personal/open-interpreter/interpreter/core/core.py", line 151, in _respond
yield from respond(self)
File "/Volumes/T7/WorkSpace/personal/open-interpreter/interpreter/core/respond.py", line 46, in respond
for chunk in interpreter._llm(messages_for_llm):
File "/Volumes/T7/WorkSpace/personal/open-interpreter/interpreter/core/llm/setup_openai_coding_llm.py", line 79, in coding_llm
messages = tt.trim(
File "/opt/anaconda3/envs/open-interpreter/lib/python3.10/site-packages/tokentrim/tokentrim.py", line 189, in trim
shorten_message_to_fit_limit(message, tokens_remaining, model)
File "/opt/anaconda3/envs/open-interpreter/lib/python3.10/site-packages/tokentrim/tokentrim.py", line 95, in shorten_message_to_fit_limit
new_length = int(len(encoding.encode(content)) * ratio)
File "/opt/anaconda3/envs/open-interpreter/lib/python3.10/site-packages/tiktoken/core.py", line 116, in encode
if match := _special_token_regex(disallowed_special).search(text):
TypeError: expected string or buffer
using multiple prompts didn't help. Tried a few python libraries and default preview app
Here, this search method's argument should be string or buffer.
But When I got an error, it was list type. It's a reason.
This is a bug in the program.
I'll try to fix it.
https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7dd73562b5e211d728723d/tiktoken/core.py#L113-L117
if disallowed_special:
if not isinstance(disallowed_special, frozenset):
disallowed_special = frozenset(disallowed_special)
if match := _special_token_regex(disallowed_special).search(text):
raise_disallowed_special_token(match.group())
Problem is here.
Type "image" in message create "array" content as new_message, but above code I wrote doesn't allow array.
This happens in only python version.
https://github.com/KillianLucas/open-interpreter/blob/05daedee6279637f1e98ad5737b4b604a8e3be3c/interpreter/core/utils/convert_to_openai_messages.py#L68-L100
It's the reason that terminal version doesn't get this error, it has exception handling.
If we add codes like this for python one, it may fix.
I'll check it after tomorrow.
https://github.com/KillianLucas/open-interpreter/blob/05daedee6279637f1e98ad5737b4b604a8e3be3c/interpreter/core/llm/setup_text_llm.py#L64-L106
I checkd codes but gpt-4-vision-preview was not even set in the python version.
There is in Roadmap haha.
Local and vision should be reserved for TUI, more granular settings for Python
I tried to update it but it's not so simple task.
Is there any project branch of it??
@KillianLucas @Notnaton