[optim][ub] pt2 values keep flip-flopping nondeterministically

[github-actions[bot]]

TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 806d1a871ddfd2d38e1791489892009feaec8425

Affected PyTorch commit: 3316374d14f2e488b65dfb4d0a2560d8414fa19f

Affected Tests:

• timm_vision_transformer_large, Adadelta, cuda, (pt2) no_foreach: +482341.57958%
• timm_vision_transformer_large, RAdam, cuda, differentiable: +81.55424%
• timm_vision_transformer, AdamW, cuda, (pt2) no_foreach: -99.98224%
• timm_vision_transformer, AdamW, cuda, (pt2) foreach: -66.83298%
• compile_time, timm_vision_transformer, AdamW, cuda, (pt2) no_foreach: +1891.78438%
• compile_time, timm_vision_transformer, SGD, cuda, (pt2) foreach, momentum=0.9, nesterov: +895.46175%
• compile_time, timm_vision_transformer, RMSprop, cuda, (pt2) foreach: +2183.02906%
• DALLE2_pytorch, AdamW, cuda, fused, capturable, amsgrad: +38.77857%
• resnet18, Adadelta, cuda, (pt2) no_foreach: +735169.67855%
• resnet18, AdamW, cuda, (pt2) no_foreach: +606511.29533%
• resnet18, AdamW, cuda, (pt2) foreach: +192.09611%
• compile_time, resnet18, Adadelta, cuda, (pt2) foreach: +1824.75319%
• compile_time, resnet18, Adagrad, cuda, (pt2) foreach: +2646.24568%
• compile_time, resnet18, AdamW, cuda, (pt2) foreach: +1720.93538%

Tests that were no longer run on affected commit:

Tests that were newly added on affected commit:

Runtime regressions found? An errors log was found. Please investigate runtime errors by looking into the logs of the workflow linked.

GitHub workflow that triggered this issue: https://github.com/pytorch/benchmark/actions/runs/6141749176

cc @janeyx99