joshpopelka20gmail

Results 3 comments of joshpopelka20gmail

[Feature] Implementation of multi-gpu KV cache (RingAttention)

Thanks for the help! That's it at the moment. I'm just trying to implement the first part of the algorithm, which is the sequence parallelism. As I work through it,...

Cross GPU device mapping feature

Sorry, may not have been clear. I got that error when running with the flag set to '-fPIC'. I haven't tried running the code using the git repo and cargo....

Enabling prefix cache for llama3 gguf

This is excellent news! I'm looking forward to testing the new feature when you are finished. Thank you for taking the time to work on it. I was thinking I...