gpt4all
gpt4all copied to clipboard
Published
20 hours ago •
nomic-ai
nomic-ai
GPT4ALL API Unable to instantiate model: Model format not supported gguf
While there are other issues open that suggest the same error, ultimately it doesn't seem that this issue was fixed. Operating on the most recent version of gpt4all as well as most recent python bindings from pip. This error happens with all gguf models on Mac. The models will load using the GUI, but will not work with docker api. On top of this the API managed by the App (GUI) doesn't actually boot up. The api server setting in the settings doesn't do anything.
Models were first manually added to model directory for the api but that didn't work so then I added to the .env file to attempt having it download. Both situations fail.
(gpt4all) gpt4all/gpt4all-api$ docker compose up --build
[+] Running 1/0
⠿ Container gpt4all_api Created 0.0s
Attaching to gpt4all_api
gpt4all_api | Checking for script in /app/prestart.sh
gpt4all_api | There is no script /app/prestart.sh
gpt4all_api | INFO: Will watch for changes in these directories: ['/app']
gpt4all_api | WARNING: "workers" flag is ignored when reloading is enabled.
gpt4all_api | INFO: Uvicorn running on http://0.0.0.0:4891 (Press CTRL+C to quit)
gpt4all_api | INFO: Started reloader process [1] using WatchFiles
gpt4all_api | INFO: Started server process [13]
gpt4all_api | INFO: Waiting for application startup.
gpt4all_api | [2024-02-13 21:49:23,729 13:MainThread] api_v1.events - INFO -
gpt4all_api | Starting up GPT4All API
gpt4all_api | | events.py:22
gpt4all_api | [2024-02-13 21:49:23,730 13:MainThread] main - INFO - Downloading/fetching model: /models/mistral-7b-instruct-v0.1.Q4_0 | main.py:37
gpt4all_api | LLModel ERROR: CPU does not support AVX
gpt4all_api | ERROR: Traceback (most recent call last):
gpt4all_api | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 734, in lifespan
gpt4all_api | async with self.lifespan_context(app) as maybe_state:
gpt4all_api | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 610, in __aenter__
gpt4all_api | await self._router.startup()
gpt4all_api | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 711, in startup
gpt4all_api | await handler()
gpt4all_api | File "/app/main.py", line 40, in startup
gpt4all_api | model = GPT4All(model_name=settings.model, model_path=settings.gpt4all_path)
gpt4all_api | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api | File "/usr/local/lib/python3.11/site-packages/gpt4all/gpt4all.py", line 102, in __init__
gpt4all_api | self.model = _pyllmodel.LLModel(self.config["path"], n_ctx, ngl)
gpt4all_api | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api | File "/usr/local/lib/python3.11/site-packages/gpt4all/_pyllmodel.py", line 178, in __init__
gpt4all_api | raise ValueError(f"Unable to instantiate model: {'null' if s is None else s.decode()}")
gpt4all_api | ValueError: Unable to instantiate model: Model format not supported (no matching implementation found)
gpt4all_api |
gpt4all_api | ERROR: Application startup failed. Exiting.
@cebtenzzre so far not seeing any difference by utilizing that PR. The error persists...
Is there something in particular or a strong reasoning for suggesting that #1729 would help?
⠋ gpt4all_api The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
Attaching to gpt4all_api
gpt4all_api | Checking for script in /app/prestart.sh
gpt4all_api | There is no script /app/prestart.sh
gpt4all_api | INFO: Will watch for changes in these directories: ['/app']
gpt4all_api | WARNING: "workers" flag is ignored when reloading is enabled.
gpt4all_api | INFO: Uvicorn running on http://0.0.0.0:4891 (Press CTRL+C to quit)
gpt4all_api | INFO: Started reloader process [1] using WatchFiles
100%|██████████| 45.9M/45.9M [00:02<00:00, 18.4MiB/s]
gpt4all_api | LLModel ERROR: CPU does not support AVX
gpt4all_api | Process SpawnProcess-1:
gpt4all_api | Traceback (most recent call last):
gpt4all_api | File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
gpt4all_api | self.run()
gpt4all_api | File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
gpt4all_api | self._target(*self._args, **self._kwargs)
gpt4all_api | File "/usr/local/lib/python3.11/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started
gpt4all_api | target(sockets=sockets)
gpt4all_api | File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 59, in run
gpt4all_api | return asyncio.run(self.serve(sockets=sockets))
gpt4all_api | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api | File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
gpt4all_api | return runner.run(main)
gpt4all_api | ^^^^^^^^^^^^^^^^
gpt4all_api | File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
gpt4all_api | return self._loop.run_until_complete(task)
gpt4all_api | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api | File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
gpt4all_api | File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 66, in serve
gpt4all_api | config.load()
gpt4all_api | File "/usr/local/lib/python3.11/site-packages/uvicorn/config.py", line 471, in load
gpt4all_api | self.loaded_app = import_from_string(self.app)
gpt4all_api | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api | File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string
gpt4all_api | module = importlib.import_module(module_str)
gpt4all_api | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api | File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
gpt4all_api | return _bootstrap._gcd_import(name[level:], package, level)
gpt4all_api | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api | File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
gpt4all_api | File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
gpt4all_api | File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
gpt4all_api | File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
gpt4all_api | File "<frozen importlib._bootstrap_external>", line 940, in exec_module
gpt4all_api | File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
gpt4all_api | File "/app/main.py", line 16, in <module>
gpt4all_api | from api_v1.api import router as v1_router
gpt4all_api | File "/app/api_v1/api.py", line 2, in <module>
gpt4all_api | from .routes import chat, completions, embeddings, engines, health
gpt4all_api | File "/app/api_v1/routes/embeddings.py", line 62, in <module>
gpt4all_api | embedder = Embed4All(model_path=settings.gpt4all_path)
gpt4all_api | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api | File "/home/appuser/.local/lib/python3.11/site-packages/gpt4all/gpt4all.py", line 44, in __init__
gpt4all_api | self.gpt4all = GPT4All(model_name or 'all-MiniLM-L6-v2-f16.gguf', n_threads=n_threads, **kwargs)
gpt4all_api | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api | File "/home/appuser/.local/lib/python3.11/site-packages/gpt4all/gpt4all.py", line 102, in __init__
gpt4all_api | self.model = _pyllmodel.LLModel(self.config["path"], n_ctx, ngl)
gpt4all_api | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api | File "/home/appuser/.local/lib/python3.11/site-packages/gpt4all/_pyllmodel.py", line 178, in __init__
gpt4all_api | raise ValueError(f"Unable to instantiate model: {'null' if s is None else s.decode()}")
gpt4all_api | ValueError: Unable to instantiate model: Model format not supported (no matching implementation found)
identical issue for me on gpt4all v2.2.1.post1
Running on a M3 Macbook macOS 14 Sonoma. Though my Python, and presumably everything else related to it, is running under x86 rosetta emulation.
Though my Python, and presumably everything else related to it, is running under x86 rosetta emulation.
Is there any reason you can't run the native arm64 version of GPT4All on your Mac? It has Metal support and gets much more testing than the x86_64 version.