jan icon indicating copy to clipboard operation
jan copied to clipboard

Published 20 hours ago • verified publisher icon janhq

bug: Load model failed with error TypeError: fetch failed

Open Graninius opened this issue 11 months ago • 3 comments

Can't start any model.

❯ tail -n 50 ~/jan/logs/app.log

2024-03-19T09:21:03.387Z [NITRO]::Debug: Loading model with params {"ctx_len":4096,"prompt_template":"<|system|>\n{system_message}<|user|>\n{prompt}<|assistant|>","llama_model_path":"/home/feltsi/jan/models/tinyllama-1.1b/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf","system_prompt":"<|system|>\n","user_prompt":"<|user|>\n","ai_prompt":"<|assistant|>","cpu_threads":4,"ngl":100}
2024-03-19T09:21:03.699Z [NITRO]::Debug: Nitro exited with code: null
2024-03-19T09:21:05.209Z [NITRO]::Error: Load model failed with error TypeError: fetch failed
2024-03-19T09:21:05.209Z [NITRO]::Error: TypeError: fetch failed
2024-03-19T09:21:05.238Z [APP]::Error: Failed to fetch
2024-03-19T09:21:37.217Z [NITRO]::CPU informations - 4
2024-03-19T09:21:37.218Z [NITRO]::Debug: Request to kill Nitro
2024-03-19T09:21:37.222Z [NITRO]::Debug: Nitro process is terminated
2024-03-19T09:21:37.223Z [NITRO]::Debug: Spawning Nitro subprocess...
2024-03-19T09:21:37.224Z [NITRO]::Debug: Spawn nitro at path: /home/user/jan/extensions/@janhq/inference-nitro-extension/dist/bin/linux-cpu/nitro, and args: 1,127.0.0.1,3928
2024-03-19T09:21:37.237Z [NITRO]::Debug:
09:21:37.536Z [NITRO]::Debug: Nitro is ready
2024-03-19T09:21:37.536Z [NITRO]::Debug: Loading model with params {"ctx_len":4096,"prompt_template":"<|system|>\n{system_message}<|user|>\n{prompt}<|assistant|>","llama_model_path":"/home/user/jan/models/tinyllama-1.1b/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf","system_prompt":"<|system|>\n","user_prompt":"<|user|>\n","ai_prompt":"<|assistant|>","cpu_threads":4,"ngl":100}
2024-03-19T09:21:37.865Z [NITRO]::Debug: Nitro exited with code: null
2024-03-19T09:21:39.375Z [NITRO]::Error: Load model failed with error TypeError: fetch failed
2024-03-19T09:21:39.375Z [NITRO]::Error: TypeError: fetch failed
2024-03-19T09:21:54.009Z [NITRO]::Debug: Request to kill Nitro
2024-03-19T09:21:54.083Z [NITRO]::Debug: Nitro process is terminated
2024-03-19T16:53:31.904Z [APP]::Version: 0.4.9
2024-03-19T16:54:13.407Z [NITRO]::CPU informations - 4
2024-03-19T16:54:13.410Z [NITRO]::Debug: Request to kill Nitro
2024-03-19T16:54:13.521Z [NITRO]::Debug: Nitro process is terminated
2024-03-19T16:54:13.524Z [NITRO]::Debug: Spawning Nitro subprocess...
2024-03-19T16:54:13.525Z [NITRO]::Debug: Spawn nitro at path: /home/user/jan/extensions/@janhq/inference-nitro-extension/dist/bin/linux-cpu/nitro, and args: 1,127.0.0.1,3928
2024-03-19T16:54:13.548Z [NITRO]::Debug:
16:54:13.847Z [NITRO]::Debug: Nitro is ready
2024-03-19T16:54:13.848Z [NITRO]::Debug: Loading model with params {"ctx_len":4096,"prompt_template":"<|system|>\n{system_message}<|user|>\n{prompt}<|assistant|>","llama_model_path":"/home/feltsi/jan/models/tinyllama-1.1b/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf","system_prompt":"<|system|>\n","user_prompt":"<|user|>\n","ai_prompt":"<|assistant|>","cpu_threads":4,"ngl":100}
2024-03-19T16:54:14.436Z [NITRO]::Debug: Nitro exited with code: null
2024-03-19T16:54:15.950Z [NITRO]::Error: Load model failed with error TypeError: fetch failed
2024-03-19T16:54:15.950Z [NITRO]::Error: TypeError: fetch failed

Manjaro KDE, unstable, i7-3740QM, Intel + AMD Hd8750M, 16Gb Ram

Graninius avatar Mar 19 '24 17:03 Graninius

I'm having the same issue

Running version 0.4.9 on a Windows laptop, 32GB RAM, RTX 3060, 6GB VRAM, i7-12700H

2024-03-20T08:50:42.703Z [NITRO]::CPU informations - 15
2024-03-20T08:50:42.704Z [NITRO]::Debug: Request to kill Nitro
2024-03-20T08:50:42.706Z [NITRO]::Debug: Nitro process is terminated
2024-03-20T08:50:43.214Z [NITRO]::Debug: Spawn nitro at path: C:\Users\MyUser\jan\extensions\@janhq\inference-nitro-extension\dist\bin\win-cuda-12-0\nitro.exe, and args: 1,127.0.0.1,3928
2024-03-20T08:50:43.214Z [NITRO]::Debug: Spawning Nitro subprocess...
2024-03-20T08:50:43.300Z [NITRO]::Debug: [93m [94m [93m [94m [93m [94m [93m_[94m_[93m_[94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m_[94m_[93m_[94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m_[94m_[93m_[94m [93m [94m [93m [94m [0m
[93m [94m [93m [94m [93m [94m/[93m_[94m_[93m/[94m [93m [94m [93m [94m [93m [94m [93m [94m_[93m_[94m_[93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m_[93m_[94m_[93m [94m [93m [94m [93m [94m [93m [94m [93m/[94m [93m [94m/[93m\[94m [93m [94m [93m [94m [93m [94m [93m [94m [93m/[94m [93m [94m/[93m\[94m [93m [94m [93m [0m
[93m [94m [93m [94m [93m [94m\[93m [94m [93m\[94m:[93m\[94m [93m [94m [93m [94m [93m [94m/[93m [94m [93m/[94m\[93m [94m [93m [94m [93m [94m [93m [94m [93m [94m/[93m [94m [93m/[94m\[93m [94m [93m [94m [93m [94m [93m/[94m [93m [94m/[93m:[94m:[93m\[94m [93m [94m [93m [94m [93m [94m [93m/[94m [93m [94m/[93m:[94m:[93m\[94m [93m [94m [0m
[93m [94m [93m [94m [93m [94m [93m\[94m [93m [94m\[93m:[94m\[93m [94m [93m [94m [93m/[94m [93m [94m/[93m:[94m/[93m [94m [93m [94m [93m [94m [93m [94m [93m/[94m [93m [94m/[93m:[94m/[93m [94m [93m [94m [93m [94m/[93m [94m [93m/[94m:[93m/[94m\[93m:[94m\[93m [94m [93m [94m [93m [94m/[93m [94m [93m/[94m:[93m/[94m\[93m:[94m\[93m [94m [0m
[93m [94m [93m_[94m_[93m_[94m_[93m_[94m\[93m_[94m_[93m\[94m:[93m\[94m [93m [94m/[93m_[94m_[93m/[94m:[93m:[94m\[93m [94m [93m [94m [93m [94m [93m [94m/[93m [94m [93m/[94m:[93m/[94m [93m [94m [93m [94m [93m/[94m [93m [94m/[93m:[94m/[93m [94m [93m\[94m:[93m\[94m [93m [94m [93m/[94m [93m [94m/[93m:[94m/[93m [94m [93m\[94m:[93m\[94m [0m
[93m [94m/[93m_[94m_[93m/[94m:[93m:[94m:[93m:[94m:[93m:[94m:[93m:[94m\[93m [94m\[93m_[94m_[93m\[94m/[93m\[94m:[93m\[94m_[93m_[94m [93m [94m [93m/[94m [93m [94m/[93m:[94m:[93m\[94m [93m [94m [93m [94m/[93m_[94m_[93m/[94m:[93m/[94m [93m/[94m:[93m/[94m_[93m_[94m_[93m [94m/[93m_[94m_[93m/[94m:[93m/[94m [93m\[94m_[93m_[94m\[93m:[94m\[0m
[93m [94m\[93m [94m [93m\[94m:[93m\[94m~[93m~[94m\[93m~[94m~[93m\[94m/[93m [94m [93m [94m [93m\[94m [93m [94m\[93m:[94m\[93m/[94m\[93m [94m/[93m_[94m_[93m/[94m:[93m/[94m\[93m:[94m\[93m [94m [93m [94m\[93m [94m [93m\[94m:[93m\[94m/[93m:[94m:[93m:[94m:[93m:[94m/[93m [94m\[93m [94m [93m\[94m:[93m\[94m [93m/[94m [93m [94m/[93m:[94m/[0m
[93m [94m [93m\[94m [93m [94m\[93m:[94m\[93m [94m [93m~[94m~[93m~[94m [93m [94m [93m [94m [93m [94m\[93m_[94m_[93m\[94m:[93m:[94m/[93m [94m\[93m_[94m_[93m\[94m/[93m [94m [93m\[94m:[93m\[94m [93m [94m [93m\[94m [93m [94m\[93m:[94m:[93m/[94m~[93m~[94m~[93m~[94m [93m [94m [93m\[94m [93m [94m\[93m:[94m\[93m [94m [93m/[94m:[93m/[94m [0m
[93m [94m [93m [94m\[93m [94m [93m\[94m:[93m\[94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m/[93m_[94m_[93m/[94m:[93m/[94m [93m [94m [93m [94m [93m [94m [93m\[94m [93m [94m\[93m:[94m\[93m [94m [93m [94m\[93m [94m [93m\[94m:[93m\[94m [93m [94m [93m [94m [93m [94m [93m [94m\[93m [94m [93m\[94m:[93m\[94m/[93m:[94m/[93m [94m [0m
[93m [94m [93m [94m [93m\[94m [93m [94m\[93m:[94m\[93m [94m [93m [94m [93m [94m [93m [94m [93m [94m\[93m_[94m_[93m\[94m/[93m [94m [93m [94m [93m [94m [93m [94m [93m [94m\[93m_[94m_[93m\[94m/[93m [94m [93m [94m [93m\[94m [93m [94m\[93m:[94m\[93m [94m [93m [94m [93m [94m [93m [94m [93m\[94m [93m [94m\[93m:[94m:[93m/[94m [93m [94m [0m
[93m [94m [93m [94m [93m [94m\[93m_[94m_[93m\[94m/[93m [94m [93m [94m [93m [
2024-03-20T08:50:43.529Z [NITRO]::Debug: Nitro is ready
2024-03-20T08:50:43.530Z [NITRO]::Debug: Loading model with params {"ctx_len":8192,"embedding":false,"prompt_template":"<|im_start|>system\n{system_message}<|im_end|>\n<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n<|im_end|>","llama_model_path":"C:\\Users\\MyUser\\jan\\models\\starchat2-15b-v0.1-Q5_K_M\\starchat2-15b-v0.1-Q5_K_M.gguf","system_prompt":"<|im_start|>system\n","user_prompt":"<|im_end|>\n<|im_start|>user\n","ai_prompt":"<|im_end|>\n<|im_start|>assistant\n<|im_end|>","cpu_threads":15,"ngl":100}
2024-03-20T08:50:43.573Z [NITRO]::Error: ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060 Laptop GPU, compute capability 8.6, VMM: yes

2024-03-20T08:50:43.691Z [NITRO]::Debug: 94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m [93m [94m\[93m_[94m_[93m\[94m/[93m [94m [93m [94m [93m [94m [93m [94m [93m [94m\[93m_[94m_[93m\[94m/[93m [94m [93m [94m [0m
[0m20240320 08:50:43.300000 UTC 22276 INFO  Nitro version:  - main.cc:50
20240320 08:50:43.300000 UTC 22276 INFO  Server started, listening at: 127.0.0.1:3928 - main.cc:54
20240320 08:50:43.300000 UTC 22276 INFO  Please load your model - main.cc:55
20240320 08:50:43.300000 UTC 22276 INFO  Number of thread is:20 - main.cc:62
20240320 08:50:43.532000 UTC 24376 INFO  Setting up GGML CUBLAS PARAMS - llamaCPP.cc:626
{"timestamp":1710924643,"level":"INFO","function":"loadModelImpl","line":642,"message":"system info","n_threads":15,"total_threads":20,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | "}

2024-03-20T08:50:43.734Z [NITRO]::Debug: Nitro exited with code: 3221225477
2024-03-20T08:50:45.255Z [NITRO]::Error: Load model failed with error TypeError: fetch failed
2024-03-20T08:50:45.256Z [NITRO]::Error: TypeError: fetch failed
2024-03-20T08:50:47.477Z [APP]::Error: Failed to fetch

Model

{
  "object": "model",
  "version": 1,
  "format": "gguf",
  "sources": [
    {
      "url": "https://huggingface.co/bartowski/starchat2-15b-v0.1-GGUF/resolve/main/starchat2-15b-v0.1-Q5_K_M.gguf",
      "filename": "starchat2-15b-v0.1-Q5_K_M.gguf"
    }
  ],
  "id": "starchat2-15b-v0.1-Q5_K_M",
  "name": "starchat2-15b-v0.1-Q5_K_M",
  "created": 1710923692324,
  "description": "",
  "settings": {
    "ctx_len": 8192,
    "embedding": false,
    "prompt_template": "<|im_start|>system\n{system_message}<|im_end|>\n<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n<|im_end|>",
    "llama_model_path": "starchat2-15b-v0.1-Q5_K_M.gguf"
  },
  "parameters": {
    "temperature": 0.7,
    "top_p": 0.95,
    "stream": true,
    "max_tokens": 8192,
    "stop": [
      "<|im_end|>"
    ],
    "frequency_penalty": 0,
    "presence_penalty": 0
  },
  "metadata": {
    "size": 11431497952,
    "author": "User",
    "tags": []
  },
  "engine": "nitro"
}

armanfellowmind avatar Mar 20 '24 08:03 armanfellowmind

Linking related ticket: https://github.com/janhq/nitro/pull/464 https://github.com/janhq/nitro/issues/463

Please stay tuned for future update from us, thank you 🙏

Van-QA avatar Mar 21 '24 07:03 Van-QA

Hi @Graninius Could you retry with the steps I mentioned here and let us know the result please: https://github.com/janhq/jan/issues/2489 Thanks

hiro-v avatar Mar 25 '24 08:03 hiro-v