LLaVa not working correctly

[kuri54]

• Mac Studio M2 Ultra
• Python 3.12.7: Conda virtual environment
• exo: latest version

After launching exo, I accessed the Chat interface, selected LLaVa 1.5 7B, and uploaded an image. I then entered a text prompt and pressed enter, but the following error message was displayed:

Error handling request
Traceback (most recent call last):
  File
"/Users/Kurita/miniconda3/envs/exo/lib/python3.12/site-packages/aiohttp/web_protocol.py",
line 459, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/Users/Kurita/miniconda3/envs/exo/lib/python3.12/site-packages/aiohttp/web_app.py", line
537, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File
"/Users/Kurita/miniconda3/envs/exo/lib/python3.12/site-packages/aiohttp/web_middlewares.py
", line 114, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/Kurita/Desktop/exo/exo/exo/api/chatgpt_api.py", line 191, in middleware
    return await asyncio.wait_for(handler(request), timeout=self.response_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/Kurita/miniconda3/envs/exo/lib/python3.12/asyncio/tasks.py", line 520, in
wait_for
    return await fut
           ^^^^^^^^^
  File "/Users/Kurita/Desktop/exo/exo/exo/api/chatgpt_api.py", line 199, in middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/Kurita/Desktop/exo/exo/exo/api/chatgpt_api.py", line 247, in
handle_post_chat_completions
    prompt, image_str = build_prompt(tokenizer, chat_request.messages)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/Kurita/Desktop/exo/exo/exo/api/chatgpt_api.py", line 119, in build_prompt
    prompt = tokenizer.apply_chat_template([m.to_dict() for m in messages],
tokenize=False, add_generation_prompt=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/Users/Kurita/miniconda3/envs/exo/lib/python3.12/site-packages/transformers/tokenization_
utils_base.py", line 1833, in apply_chat_template
    rendered_chat = compiled_template.render(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/Users/Kurita/miniconda3/envs/exo/lib/python3.12/site-packages/jinja2/environment.py",
line 1304, in render
    self.environment.handle_exception()
  File
"/Users/Kurita/miniconda3/envs/exo/lib/python3.12/site-packages/jinja2/environment.py",
line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/Users/Kurita/miniconda3/envs/exo/lib/python3.12/site-packages/jinja2/sandbox.py",
line 392, in call
    if not __self.is_safe_callable(__obj):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/Kurita/miniconda3/envs/exo/lib/python3.12/site-packages/jinja2/sandbox.py",
line 276, in is_safe_callable
    getattr(obj, "unsafe_callable", False) or getattr(obj, "alters_data", False)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: 'list object' has no attribute 'strip'

Additionally, when I use the ChatGPT-compatible API endpoint, I encounter an error as well. The error displayed is:

500 Internal Server Error
Server got itself in trouble

Is there a bug in the current release, or could it be an issue with my setup? From what I can verify, other LLMs seem to be running fine.

[AlexCheema]

@varshith15 do you know what's up here?

[ThaddeusJiang]

I got the error message

{"detail": "Error processing prompt (see logs with DEBUG>=2): Model type llava not supported."}%    

My request is copied from README

curl http://localhost:52415/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "llava-1.5-7b-hf",
     "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What are these?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "http://images.cocodataset.org/val2017/000000039769.jpg"
            }
          }
        ]
      }
    ],
     "temperature": 0.0
   }'

tinychat screenshot

[JonathanPorta]

I am also receiving this error but I can't seem to increase the verbosity of the output using the debug flag.

[JackPala]

I also have this issue

[Evanev7]

Shouldn't be relevant in 1.0, please reopen if it's still causing problems!