My local RAG return garbled text

[chengr4]

Hi,

I followed steps in README but failed at the latest step. The return from the prompt is garbled text.

Wish anyone can guide me to the right path. 🫠

But the server's search seems to be correct? 🤔

Have Done

• [x] Built WASM binary
• [x] Started qdrant
• [x] Fed paris.txt

[juntao]

Can you paste the command you used to start the server?

[chengr4]

I copy-paste the one in README

wasmedge --dir .:. --nn-preload default:GGML:AUTO:Llama-2-7b-chat-hf-Q5_K_M.gguf \
    --nn-preload embedding:GGML:AUTO:all-MiniLM-L6-v2-ggml-model-f16.gguf \
    rag-api-server.wasm \
    --model-name Llama-2-7b-chat-hf-Q5_K_M,all-MiniLM-L6-v2-ggml-model-f16 \
    --ctx-size 4096,384 \
    --prompt-template llama-2-chat \
    --rag-prompt "Use the following pieces of context to answer the user's question.\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\n----------------\n" \
    --log-prompts \
    --log-stat

[juntao]

Can you add --model-alias default,embedding to the command and try again? Thanks!

[chengr4]

Look no difference 😢.

• Run server 1:54
• Output: 3:35

Video link: https://drive.google.com/file/d/1OXLZhQwcyabCpgQ8_YXnLrGRJNl-81-N/view?usp=sharing

[chengr4]

Retry on 2024.06.15

stuck by running

wasmedge --dir .:. --nn-preload default:GGML:AUTO:Llama-2-7b-chat-hf-Q5_K_M.gguf \
    --nn-preload embedding:GGML:AUTO:all-MiniLM-L6-v2-ggml-model-f16.gguf \
    rag-api-server.wasm \
    --model-name Llama-2-7b-chat-hf-Q5_K_M,all-MiniLM-L6-v2-ggml-model-f16 \
    --ctx-size 4096,384 \
    --prompt-template llama-2-chat,embedding \
    --rag-prompt "Use the following pieces of context to answer the user's question.\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\n----------------\n" \
    --log-prompts \
    --log-stat

Get error: "wasi-logging plugin not installed. Please install the plugin and restart WasmEdge."

[2024-06-15 15:07:40.370] [error] wasi-logging plugin not installed. Please install the plugin and restart WasmEdge.
[2024-06-15 15:07:40.371] [error] execution failed: host function failed, Code: 0x40e
[2024-06-15 15:07:40.371] [error]     When executing function name: "_start"

but if I install wasi-logging, wasi_nn will be removed

---

Version rag: 0.6.6 wasmedge: 0.14.0

[juntao]

You can install both plugins. Just re-run the installer. It will auto install both.

curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install_v2.sh | bash -s -- -v 0.13.5

[chengr4]

Thanks for guidance.

However, I still got garbled text as before. 😢

---

Version rag: 0.6.6 wasmedge: 0.13.5

[apepkuss]

@chengr4 Please update --prompt-template llama-2-chat to --prompt-template llama-2-chat,embedding.

[chengr4]

i ran:

wasmedge --dir .:. --nn-preload default:GGML:AUTO:Llama-2-7b-chat-hf-Q5_K_M.gguf \
    --nn-preload embedding:GGML:AUTO:all-MiniLM-L6-v2-ggml-model-f16.gguf \
    rag-api-server.wasm \
    --model-name Llama-2-7b-chat-hf-Q5_K_M,all-MiniLM-L6-v2-ggml-model-f16 \
    --ctx-size 4096,384 \
    --prompt-template llama-2-chat,embedding \
    --rag-prompt "Use the following pieces of context to answer the user's question.\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\n----------------\n" \
    --log-prompts \
    --log-stat

However, I still got garbled

---

Version rag: 0.7.1 wasmedge: 0.13.5