web-llm

web-llm copied to clipboard

I noticed that the readme mentioned that we need "6.4G GPU memory" to run the demo. However, my Mac Pro only has a 4GB memory, just wondering whether there is any approaches to run on a 4GB GPU memory PC? Thanks!

Same issue, I got 4g dedicated GPU memory, and 16g shared, but inited failed. Any workaround? Thanks

It's coming soon. Our team is testing running vicuna within 4gb memory internally and will make it public soon

It's coming soon. Our team is testing running vicuna within 4gb memory internally and will make it public soon

Thanks. I thought it is hard and maybe impossible. Looking forward to the 4gb memory one. It will be super meaningful.

Try out our latest project https://github.com/mlc-ai/mlc-llm. You can run a model within 4gb memory constraints in native runtime. We will support 4gb llm on web later

Try out our latest project https://github.com/mlc-ai/mlc-llm. You can run a model within 4gb memory constraints in native runtime. We will support 4gb llm on web later

Thanks. Since it does not need web browser and can be run native on terminal. I tried it on a linux server and the model works by following the instruction. Thanks!