mistral-inference
mistral-inference copied to clipboard
Fix device error when using cuda device other than cuda:0
Fixes https://github.com/mistralai/mistral-inference/issues/215
Attention bias was being created on cuda:0 regardless of the selected cuda device as the correct device was not being passed to from_seqlens() in BufferCache.get_input_metadata()