Doesn't see Falcon 40b converted, to be a GGML file..ctransformers problem?

[linuxmagic-mp]

On Ubuntu Focal..

Current binding: c_transformers
Current model: None
Menu:
1 - WizardLM-Uncensored-Falcon-40b.reference
2 - Install model
3 - Change binding
4 - Back
Enter your choice: 1
You selected model: WizardLM-Uncensored-Falcon-40b.reference
Loading a reference model:
/home/michael/models/WizardLM-Uncensored-Falcon-40b
error loading model: unknown (magic, version) combination: 04034b50, 08080000; is this really a GGML file?
falcon_init_from_file: failed to load model

I have both the original, as well as the converted, and quantized versions in that directory.

drwxrwxr-x 1 michael michael         1206 Jul 22 07:27 .
drwxrwxr-x 1 michael michael          140 Jul 21 20:04 ..
-rw-rw-r-- 1 michael michael          719 Jul 21 20:04 config.json
-rw-rw-r-- 1 michael michael         2511 Jul 21 20:04 configuration_RW.py
-rw-rw-r-- 1 michael michael          111 Jul 21 20:04 generation_config.json
-rw-rw-r-- 1 michael michael 167344901889 Jul 22 07:21 ggml-model-falcon-40b-wizardlm-f32.bin
-rw-rw-r-- 1 michael michael  25372601600 Jul 22 07:30 ggml-model-falcon-40b-wizardlm-qt_k_m.bin
drwxrwxr-x 1 michael michael          182 Jul 22 06:57 .git
-rw-rw-r-- 1 michael michael         1477 Jul 21 20:04 .gitattributes
-rw-rw-r-- 1 michael michael           14 Jul 21 20:04 latest
-rw-rw-r-- 1 michael michael        47131 Jul 21 20:04 modelling_RW.py
-rw-rw-r-- 1 michael michael   9504786525 Jul 21 22:01 pytorch_model-00001-of-00009.bin
-rw-rw-r-- 1 michael michael   9513159151 Jul 21 22:01 pytorch_model-00002-of-00009.bin
-rw-rw-r-- 1 michael michael   9513159151 Jul 21 22:02 pytorch_model-00003-of-00009.bin
-rw-rw-r-- 1 michael michael   9513159151 Jul 21 22:01 pytorch_model-00004-of-00009.bin
-rw-rw-r-- 1 michael michael   9513159151 Jul 21 22:02 pytorch_model-00005-of-00009.bin
-rw-rw-r-- 1 michael michael   9513159151 Jul 21 22:02 pytorch_model-00006-of-00009.bin
-rw-rw-r-- 1 michael michael   9513159151 Jul 21 22:02 pytorch_model-00007-of-00009.bin
-rw-rw-r-- 1 michael michael   9513159151 Jul 21 22:02 pytorch_model-00008-of-00009.bin
-rw-rw-r-- 1 michael michael   7575236661 Jul 21 22:21 pytorch_model-00009-of-00009.bin
-rw-rw-r-- 1 michael michael        39264 Jul 21 20:04 pytorch_model.bin.index.json
-rw-rw-r-- 1 michael michael         1051 Jul 21 20:04 README.md
-rw-rw-r-- 1 michael michael          305 Jul 21 20:04 special_tokens_map.json
-rw-rw-r-- 1 michael michael          207 Jul 21 20:04 tokenizer_config.json
-rw-rw-r-- 1 michael michael      2734597 Jul 21 20:04 tokenizer.json
-rw-rw-r-- 1 michael michael        13129 Jul 21 20:04 trainer_state.json
-rw-rw-r-- 1 michael michael         5051 Jul 21 22:02 training_args.bin
-rw-rw-r-- 1 michael michael        24043 Jul 21 20:04 zero_to_fp32.py

Easy fix? Suggest that when the 'path' is chosen, if it doesn't see a GGML file, that maybe it can prompt for the actual name of the GGML file of choice.

Couldn't load model. Please verify your configuration file at /home/michael/Documents/lollms/configs or use the next menu to select a valid model
Binding returned this exception : Failed to create LLM 'falcon' from '/home/michael/models/WizardLM-Uncensored-Falcon-40b/training_args.bin'.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/lollms/app.py", line 75, in load_model
    model = ModelBuilder(self.binding).get_model()
  File "/usr/local/lib/python3.10/dist-packages/lollms/binding.py", line 285, in __init__
    self.build_model()
  File "/usr/local/lib/python3.10/dist-packages/lollms/binding.py", line 288, in build_model
    self.model = self.binding.build_model()
  File "/home/michael/Documents/lollms/bindings_zoo/c_transformers/__init__.py", line 114, in build_model
    self.model = AutoModelForCausalLM.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/ctransformers/hub.py", line 157, in from_pretrained
    return LLM(
  File "/usr/local/lib/python3.10/dist-packages/ctransformers/llm.py", line 214, in __init__
    raise RuntimeError(
RuntimeError: Failed to create LLM 'falcon' from '/home/michael/models/WizardLM-Uncensored-Falcon-40b/training_args.bin'.

personal_models_path: /home/michael/Documents/lollms/models
Binding name:c_transformers
Model name:WizardLM-Uncensored-Falcon-40b.reference
Please select a valid model or install a new one from a url

It isn't clear from the errors, but it is actually bubbling up from ctransformers, and have filed ticket there as well.

Issues with ctransformers_llm_create and Falcon-40b model #60

Only posting here to see if perhaps it is not called with correct parameters, but it 'appears' this isn't a problem with that.
Ticketing here for others trying to use ctransformers with Falcon for now.

[linuxmagic-mp]

Okay, the problem here appears to lie in the way AutoModelForCausalLM is called.. When using a ctransformers method, we should be calling it with a model_file argument. See https://github.com/marella/ctransformers/issues/60. Otherwise, it can return any .bin from the path, including training_args.bin, which of course will fail ggml checks.

[linuxmagic-mp]

And to continue, I have the actual filename in the reference file

cat ~/lollms_data/models/llama_cpp_official/falcon-40b-ggml.reference

~/models/WizardLM-Uncensored-Falcon-40b/ggml-model-falcon-40b-wizardlm-qt_k5.bin

Howeer, it doesn't see that it properly sees that as a file. Instead, earlier that was parsed down to model_path, and model_name was not preserved.. So when we run.. _find_model_path_from_dir, the optional filename is not passed in. AutoModelForCausalLM.from_pretrained(), which was called after get_model_path

if self.config.model_name.endswith(".reference"):
     with open(str(self.lollms_paths.personal_models_path / f"{self.binding_folder_name}/{self.config.model_name}"), 'r') as f:
                model_path = Path(f.read()

That means if you have the file name in a .reference it will not work. ?

[linuxmagic-mp]

Well, guess I am still talking to myself.. ;) Went up and down this code, and there doesn't seem to be a 'recommended' way to handle setting the actual filename from the config, for the ctransformers binding (or any binding). I think the work is all about pulling down the needed models, but when you have a directory of different quantized models you are kind of hooped.

The transformers binding 'should' be doing more that the simple get_model_path(self), which in itself could use a little validation.It should only read the .reference file once if it exists, and set both path and filename if the .reference file contains a fullpath to a model.

Not really sure why the transformer bindings were decided that they should NOT be part of the library itself, and placed in the users lollms personal settings.. Seems like a strange design choice.

/c_transformers/init.py needs to be updated. .

To get on with things, since noone is commenting, no use making a pull request.. I just hacked get_model_path() to set self.model_file, and the zoo.. init.py, to pass that in when it calls AutoModelForCasual. NOTE, this would ALSO work if get_model_path could return whatever is in the .reference file, but that goes against my grain, as the function could be used elsewhere, where it really does need the 'path'

[ParisNeo]

Hi there, I am sorry. I has loads of issues opened and yours was down and I was both in vacation, then was coding so i didn't see it. I have dixed your problem. It was just about the code that determines the type of the file out of its name. Now Wizard stuff is considered llama so all of his code is now accepted as llama.