jianan-gu

Hi @TimDettmers, Could you please have a review on this PR for any comment/suggestion. Thanks!

> Thank you so much for this contribution. We discussed internally how to best integrate this and other libraries. We think it is best to abstract the `extern C` interface...

> The concept and idea looks good here, and gels with [my discussion comment about splitting into multiple Python packages](https://github.com/TimDettmers/bitsandbytes/discussions/990#discussioncomment-8331515). > > I think we should refine the design a...

> Thanks, this is looking better 😄 A handful of comments within to make this more future-proof... Thanks for your kind and valuable reviews, here have refined this PR accordingly...

> Much better, thank you for putting up with my suggestions 😂 > > We'll need to come back to giving the rest of the backend methods types and docstrings,...

> Thanks for your great work @jianan-gu @akx and all ! Indeed it would be great to add nice docstrings as we can now autogenerate API docs from docstrings thanks...

> Ok, so I did a thorough review and everything still looks very good. Other than some cosmetic stuff, the one thing that I think is important is to not...

@molly-smith Hi please take a review~ thanks!

cc @jgong5 @EikanWang

> Does it make sense to also update [docs/_tutorials/automatic-tensor-parallism.md](https://github.com/microsoft/DeepSpeed/blob/master/docs/_tutorials/automatic-tensor-parallelism.md) to include this model in supported list? Have added "llama" to the doc. Thanks.