koila icon indicating copy to clipboard operation
koila copied to clipboard

Maths domain error

Open adnana-dev opened this issue 1 year ago • 1 comments

I am using Koila to solve an OOM error during my training. But the following error occurs : ``Traceback (most recent call last): File "/mnt/sdb2/Adama/configure_docker_for_transvw/pytorch/train.py", line 92, in loss.backward() File "/home/nanaa/.local/lib/python3.10/site-packages/koila/lazy.py", line 435, in backward for mini_batch_size in gpus.split_batch( File "/home/nanaa/.local/lib/python3.10/site-packages/koila/gpus.py", line 100, in split_batch batch_size = 2 ** (math.floor(math.log2(max_batch))) ValueError: math domain error``` Probably due to the value of max_batch ?

adnana-dev avatar Jul 26 '22 13:07 adnana-dev

Hi, thanks for the bug report! I'll look into what went wrong.

rentruewang avatar Jul 27 '22 09:07 rentruewang