candle icon indicating copy to clipboard operation
candle copied to clipboard

wait_until_completed is not working for metal device

Open antimora opened this issue 1 year ago • 3 comments
trafficstars

During our benchmark testing, we noticed that the Candle backend for Burn was finishing up quickly for the Metal device. Upon closer inspection, we have discovered that wait_until_completed is not functioning properly. As a fix for now, we have disabled our benchmark for the Candle Metal device (see https://github.com/tracel-ai/burn/pull/1579).

antimora avatar Apr 12 '24 21:04 antimora

I'm actually in the middle of an overhaul of the metal backend on this PR, could you try out this branch with your benchmark and see if it's still an issue? Note that I've replaced the "wait_until_completed" method with a "synchronize" method to match the cuda backend

https://github.com/huggingface/candle/pull/2037

tomsanbear avatar Apr 13 '24 14:04 tomsanbear

CC @louisfd

antimora avatar Apr 13 '24 19:04 antimora

Note that I'm also adding a synchronize method across devices in #2055, this will ensure that there is a unified and simple way to make these calls.

LaurentMazare avatar Apr 14 '24 14:04 LaurentMazare