rocm_lab bitsandbytes 0.39.0?

do u have plans for it or is it not possible rn

Jun 08 '23 18:06 ewof

The BitsAndBytes is built from https://github.com/agrocylo/bitsandbytes-rocm, which is 0.37.2 currently. It's like doing hipify and fix, but I personally don't have the knowledge nor the time to maintain an up-to-date version, so I have no plan for it at the moment.

Jun 08 '23 18:06 evshiron

I dug into this a little - the hurdle may be porting the 4-bit update over to rocm. It's also over my head but it seems like rocm can support it.

Jun 11 '23 05:06 shermdog

i have most of it ported but the .hip and .hiph files generated by hipify-clang don't work there's a lot of manual editing needed but i think it's because i don't have hipBLASLt which i have been trying to compile for a bit (their install script doesnt support arch based distros)

Jun 11 '23 05:06 ewof

I've made some solid progress on porting over 0.39.0 to rocm. It currently compiles and will load models in 4-bit but generation returns gibberish. There's still two major things to port - the wmma matrix bits and fixing bfloat16

https://github.com/TimDettmers/bitsandbytes/compare/main...shermdog:rocm_039?expand=1

Jun 15 '23 04:06 shermdog

@shermdog

I cloned your repo and tried locally. When running examples/int8_inference_huggingface.py with load_in_4bit=True, it always decoded into the same token, but load_in_8bit=True did worked.

Nice work here, and I am looking forward to your future updates!

Jun 15 '23 11:06 evshiron

Here is another fork:

https://github.com/arlo-phoenix/bitsandbytes-rocm-5.6

Which looks quite promising but I haven't tested it.

Aug 14 '23 16:08 evshiron

got load-in-4bit to work with this and textgen-webui by setting the kQuantizeBlockwise to how it is in the main repo

Sep 06 '23 02:09 ewof