text-generation-inference icon indicating copy to clipboard operation
text-generation-inference copied to clipboard

bitsandbytes: upgrade and enable CUDA Graphs for 4bit by default

Open matthewdouglas opened this issue 10 months ago • 1 comments

What does this PR do?

This PR sets the new minimum requirement for bitsandbytes to the most recent release, v0.45.0. from v0.43.0.

CUDA Graphs support for 4bit was enabled in v0.44.0, and as such this PR additionally turns this on by default for bitsandbytes-nf4 and bitsandbytes-fp4 quantization options.

The v0.45.0 release brings H100 compatibility for 8bit along with significant performance improvements.

Additionally, there are minor documentation changes, as well as some cleanup in Linear8bitLt and Linear4bit.

Fixes #2467, #780, #504

Before submitting

  • [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • [x] Did you read the contributor guideline, Pull Request section?
  • [ ] Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
  • [x] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
  • [x] Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

@OlivierDehaene @Narsil

cc: @Titus-von-Koeller

matthewdouglas avatar Dec 12 '24 22:12 matthewdouglas