Tianqi Chen
Tianqi Chen
Make sure you run npm install first to install web-llm, you would need to change the web-llm dependency to 0.2.0 instead of file as being suggested. If you have already...
try to do a fresh clone of the repo and try out commands in https://github.com/mlc-ai/web-llm/tree/main/examples/get-started
This is a known issue that we are working with the safari team to address https://bugs.webkit.org/show_bug.cgi?id=266793
That likely mean that you do not have a device that have enough GPU RAM
hmm, that seems to be sufficient, maybe you can try out https://webgpureport.org/ to see if you can find the GPU. That could be an issue of your browser, make sure...
Thank you @mwyrzykowski for enabling this, this issue can be closed. There are smaller models that we use for smaller context (the models that ends with -1k suffix) that might...
Actually @mwyrzykowski if it is possible to enable iOS to have bigger buffers, it would be nice. Since some of the 3B models can fit well on mobile, and may...
get it, thank you @mwyrzykowski for explaining, I think we can stay with 256 MB for now then, hopfully some of the the smaller models should still work well.
the latest API support token streaming via OAI style API
Unfirtunately WebGL's programming model is limited and cannot enable some fo the optimizations needed, so it likely is infeasible Good news is that Firefox is also working to land WebGPU...