pytorch_backend icon indicating copy to clipboard operation
pytorch_backend copied to clipboard

Published 20 hours ago • verified publisher icon triton-inference-server

feat: add bfloat16

Open phuhung273 opened this issue 4 months ago • 3 comments

Relates https://github.com/triton-inference-server/server/issues/7853

Update https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#datatypes after this is done

phuhung273 avatar Aug 09 '25 05:08 phuhung273

Thank you for your contribution!

This seems almost too easy. Can be as easy as just recognizing the data type? I would assume that there would be internals which need to be made aware/capable of handling the new data type. Maybe not.

Adding @tanmayv25 and @yinggeh to review/comment.

whoisj avatar Oct 09 '25 17:10 whoisj

@phuhung273 please read the Triton Contributors Contribution License Agreement. We'll need this completed prior to accepting any changes from you unless you're acting on behalf of your employer and your employer has a CCLA on file with us.

Thank you.

whoisj avatar Oct 09 '25 19:10 whoisj

This PR is a lifesaver, especially given that ONNX does not support BF16 in Triton. Torchscript and ONNX are kinda the only backend that can run complex if / else logic

MatthieuToulemont avatar Oct 20 '25 11:10 MatthieuToulemont