Kevin Eales

I came here just for this: Exact same problem on AGX Orin JP 5.1.1 L4T 35.3.1 /usr/src/llama.cpp/ggml.h(218): error: identifier "__fp16" is undefined

 Ahhhh, cortex 8+ processors no longer support neon, the library must be fully x64. They can support x32 but only when running within an x32 operating system / kernel.

@malv-c If you replace __fp16 with uint16_t on line 218 of ggml.h the project builds and cuBLAS works without issue.

Even though this successfully builds, it does seem to be attempting to use NEON, I am unsure if this will have a performance impact... ``` llama.cpp: loading model from /opt/gpt-models/vicuna-7b-1.1.ggmlv3.q8_0.bin...

Oh! This here looks like it might be the silver bullet: https://github.com/ggerganov/llama.cpp/issues/1455#issuecomment-1555761710

This might be possible using PeterL1n's virtual inference models "RobustBackgroundRemoval" and "BackgroundMatting2"

I am experiencing this as well, ``` +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC...

Hmmm it seems to go until it halts the ollama service, additionally it's lazy-loading, so the upload process doesn't begin until I supply a prompt: ``` Aug 02 12:08:12 ai-buffoli...

I would also like to take a look at this dataset if you still have it around.

I experienced this issue on a 2014 MacBook Pro on Ventura 13.6.7 (22G720); however, the mouse works just fine in windowed mode.