mistral-inference icon indicating copy to clipboard operation
mistral-inference copied to clipboard

Fix device error when using cuda device other than cuda:0

Open cornzz opened this issue 1 year ago • 0 comments

Fixes https://github.com/mistralai/mistral-inference/issues/215

Attention bias was being created on cuda:0 regardless of the selected cuda device as the correct device was not being passed to from_seqlens() in BufferCache.get_input_metadata()

cornzz avatar Aug 28 '24 12:08 cornzz