Jeffser
Alpaca Instance Error (Mistral Nemo)
Describe the bug
I discovered what appears to be a bug when attempting to run the mistral-nemo model. This error does not occur with ANY other model that I tried including original mistral. The error does not occur when using official ollama cli with the same mistral-nemo model. I tried using mistral-nemo:latest, a q4_K_M variant, and the unsloth variant from huggingface, none of which work in Alpaca. Error occurs during my first message to the model, within 1-2 seconds.
Instance Error: Message generation failed ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Expected behavior
The connection should stay alive, and mistral-nemo model should respond to me as any other model would.
Screenshots
Debugging information
llama_model_loader: loaded meta data with 35 key-value pairs and 363 tensors from /home/*user*/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-dd3af152229f92a3d61f3f115217c9c72f4b94d8be6778156dab23f894703c28 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Mistral Nemo Instruct 2407
llama_model_loader: - kv 3: general.version str = 2407
llama_model_loader: - kv 4: general.finetune str = Instruct
llama_model_loader: - kv 5: general.basename str = Mistral-Nemo
llama_model_loader: - kv 6: general.size_label str = 12B
llama_model_loader: - kv 7: general.license str = apache-2.0
llama_model_loader: - kv 8: general.languages arr[str,9] = ["en", "fr", "de", "es", "it", "pt", ...
llama_model_loader: - kv 9: llama.block_count u32 = 40
llama_model_loader: - kv 10: llama.context_length u32 = 1024000
llama_model_loader: - kv 11: llama.embedding_length u32 = 5120
llama_model_loader: - kv 12: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 13: llama.attention.head_count u32 = 32
llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 15: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 17: llama.attention.key_length u32 = 128
llama_model_loader: - kv 18: llama.attention.value_length u32 = 128
llama_model_loader: - kv 19: general.file_type u32 = 15
llama_model_loader: - kv 20: llama.vocab_size u32 = 131072
llama_model_loader: - kv 21: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 22: tokenizer.ggml.add_space_prefix bool = false
llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 24: tokenizer.ggml.pre str = tekken
llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,131072] = ["<unk>", "<s>", "</s>", "[INST]", "[...
llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,131072] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
ERROR [ollama_instances.py | generate_chat_title] ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
ERROR [ollama_instances.py | generate_response] ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
INFO [ollama_instances.py | stop] Stopping Alpaca's Ollama instance
INFO [ollama_instances.py | stop] Stopped Alpaca's Ollama instance
Traceback (most recent call last):
File "/app/share/Alpaca/alpaca/window.py", line 475, in <lambda>
self.model_dropdown.connect('notify::selected', lambda dropdown, gparam: self.global_footer.tool_selector.model_changed(dropdown))
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/app/share/Alpaca/alpaca/widgets/tools/__init__.py", line 30, in model_changed
if 'tools' in dropdown.get_selected_item().model.data.get('capabilities', ['tools']):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'model'
I am having a similar issue, have you ever found a solution?
Mistral Nemo has always been a mess. specifically that model is really inconsistent for some reason
