webllm.worker.ts:8 Device was lost. This can happen due to insufficient memory or other GPU constraints. Detailed error: [object GPUDeviceLostInfo]. Please try to reload WebLLM with a less resource-intensive model.

[jasperlan666666]

Got the 'Device was lost...' error from the new initialized chat engine but started with prior conversations. A repeatable problem.

Here is an example, 0-th and 1-th entries were from prior conversations. First conservation with new engine includes 0-th, 1-th and 2-th entries but then 'Device was lost...' problem happened.

0:{role: 'user', content: 'what is your name'} 1:{role: 'assistant', content: "Nice to meet you! My name is Ada, and I'm here to …nd helpful information. How can I help you today?"} 2:{role: 'user', content: 'what did i ask before'} 'Device was lost...' error

Anyone has any idea how to resolve?

[saquib34]

Are you seeing this on chat.webllm.ai? Perhaps try the one with -1k suffix, which has smaller kv cache, hence less memory requirement. Also try q4f16_1 instead of q4f32_1. and also The f16 error suggests that the WebGPU on your browser/device does not support f16 computation. You can check it manually at https://webgpureport.org/. If supported, you should see this shader-f16 in features:

The f16 error and Device lost error are separate. Seeing device lost with Llama3.1-q4f32_1 suggests you do not have enough RAM (it requires ^5GB according to our config.ts); seeing f16 not supported with q4f16_1 means WebGPU compatibility with f16 computation. On a side note, q4f32 models require more RAM than the q4f16 counterparts. You can see the config.ts for more.