Sergey 'Jin' Bostandzhyan

> `pipe.enable_model_cpu_offload()` > `pipe.enable_sequential_cpu_offload()` > > Can these two sentences enable you to run on an single AMD GPU? Unfortunately no, when I remove the "balanced" setting and let the...

Aha... wait, so this means that using multiple GPUs and enabling CPU offload are mutually exclusive? Is there any potential way to have CPU offload together with the use of...

> guys is there a way to use selected gpu only? Config etc? I think you can simply control the visibility of your GPUs using the `CUDA_VISIBLE_DEVICES` environment variable to...

@mpai17 no, 16GB on one card was not enough, so I ended up with OOM and gave up on this - too bad, would have loved to try it out.

I gave up on it eventually as I had to give up on many projects that were written with NVIDIA in mind and were always missing something or did not...

@Exploder98 thanks for the insight, actually Burn may indeed help here, they did not rely on foreign implementations, but they basically came up with https://github.com/tracel-ai/cubecl which is a layer over...

@mpai17 wow, thank you for sharing this! I did not know ZLUDA was still around, I thought it was kind of abandoned after AMD stopped financing it. Will try your...

The `.cargo/config.toml` worked for me, although later I moved to a fixup bbclass which would set the variable in `do_compile:prepend` as above, because that ensures builds work out of the...

> Hi, I'm just wondering if you both had any progress. Thank you in advance! > > [@UltramanKuz](https://github.com/UltramanKuz) [@jin-eld](https://github.com/jin-eld) @brayden-hai I gave up on it, LLAMA3 was just too huge...

I'd like to point out one thing though: the standalone llama.cpp repo at the current master branch (45c0e2e4c1268c2d7c8c45536f15e3c9a731ecdc) builds just fine with this command (c&p from llama.cpp build instructions) and...