AlpinDale
AlpinDale
> So just an update chunked prefill works for 2xRTX 3090 24GB NVLink for Llama 3.1 8B FP16 at full 128K context. It outputs normal non-gibberish text. Yay! Thanks alpin....
@theobjectivedad thanks for the PR and sorry for the late review! I made some changes including rebasing it to the release candidate branch. Please let me know if this is...
Hi yes, they're supported. The easiest way is to use them through Docker. I will be updating the wiki in the next release with detailed instructions. Stay tuned!
Added as of v0.6.0
I believe our p2p detection has improved as of v0.6.0. Can you try again?
Oh boy. I suppose we have to pin numpy versions too from now on. Thanks for reporting Does torch not pin numpy already? Weird.
For anyone else who might be running into this issue, please downgrade numpy: ```sh pip install numpy==1.26.4 ```
Looks like you're on python 3.12. At the moment, we only have wheels for python versions 3.8-3.11. Please downgrade. You can either use conda to do this, or a python...
Python 3.12 now officially supported as of v0.6.0
This should work perfectly fine as of v0.6.0. Feel free to re-open the issue if the problem persists.