pytorch icon indicating copy to clipboard operation
pytorch copied to clipboard

[release/2.4][ROCm][TunableOp] Fix TunableOp warmup environment variable. (#147412)

Open naromero77amd opened this issue 8 months ago • 2 comments

This PR corrects the behavior of the TunableOp warmup variables:

PYTORCH_TUNABLEOP_MAX_WARMUP_DURATION_MS
PYTORCH_TUNABLEOP_MAX_WARMUP_ITERATIONS

See the updated comments which describe how the environment variables are intended to work. Previously, if you only set one of the two environment variables the warmup iters would always be zero.

Manually tested the four possible combinations to make sure things still behavior as intended.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/147412 Approved by: https://github.com/jeffdaily

(cherry picked from commit 4b35139a462f1858cf5fe5346e16f7e1abede78d)

naromero77amd avatar Apr 03 '25 23:04 naromero77amd

This is a subtle bug that has been present for a while. I discovered it post PyTorch 2.6 release. Cherry picked from upstream and will need to be cherry picked into ROCm 2.5 and 2.6 release branches.

naromero77amd avatar Apr 03 '25 23:04 naromero77amd

Jenkins build for a9992d4cd7685b64a1ca9c419925b6bcd19de1ef commit finished as SUCCESS Links: Blue Ocean view / Build artifacts

Jenkins build for a9992d4cd7685b64a1ca9c419925b6bcd19de1ef commit finished as FAILURE Links: Blue Ocean view / Build artifacts

!cherry-pick --onto release/2.5

naromero77amd avatar Apr 21 '25 22:04 naromero77amd

Created branch autogenerated/release/2.5_cherry-pick_pr-2017 and https://github.com/ROCm/pytorch/pull/2040

rocm-mici avatar Apr 21 '25 23:04 rocm-mici

!cherry-pick --onto release/2.6

naromero77amd avatar Apr 21 '25 23:04 naromero77amd

Created branch autogenerated/release/2.6_cherry-pick_pr-2017 and https://github.com/ROCm/pytorch/pull/2042

rocm-mici avatar Apr 21 '25 23:04 rocm-mici