trt-llm-rag-windows Installed both models in Chat-With-RTX but can only use Mistral 7B int4

Installed both models in Chat-With-RTX but can only use Mistral 7B int4

Open Jason-XII opened this issue 1 year ago • 5 comments

I followed the installation instructions and the installer reported that I have successfully installed both models - Llama and Mistral. However when I started the app-launcher.bat file, it said:

Environment path found: C:\Users\Jason\AppData\Local\NVIDIA\ChatWithRTX\env_nvd_rag
App running with config
 {
    "models": {
        "supported": [
            {
                "name": "Mistral 7B int4",
                "installed": true,
                "metadata": {
                    "model_path": "model\\mistral\\mistral7b_int4_engine",
                    "engine": "llama_float16_tp1_rank0.engine",
                    "tokenizer_path": "model\\mistral\\mistral7b_hf",
                    "max_new_tokens": 1024,
                    "max_input_token": 7168,
                    "temperature": 0.1
                }
            },
            {
                "name": "Llama 2 13B int4",
                "installed": false,
                "metadata": {
                    "model_path": "model\\llama\\llama13_int4_engine",
                    "engine": "llama_float16_tp1_rank0.engine",
                    "tokenizer_path": "model\\llama\\llama13_hf",
                    "max_new_tokens": 1024,
                    "max_input_token": 3900,
                    "temperature": 0.1
                }
            }
        ],
        "selected": "Mistral 7B int4"
    },
...

It said that Llama is not installed, but it should be! What might be the possible cause of this problem? How can I fix it? Should I follow the README file instead? Extra information: I am a China user, using an (almost) new computer with a RTX 3060 GPU and 16GB of RAM.