Enable NHWC batchnorm for miopen

[dnikolaev-amd]

Environment variable PYTORCH_MIOPEN_SUGGEST_NHWC=1 enables MIOpen batchnorm for NHWC

[dnikolaev-amd]

New test addad. It compares NHWC MIOpen batchnorm with NHWC CPU batchnorm PYTORCH_MIOPEN_SUGGEST_NHWC=1 python test/test_nn.py -v TestNN.test_batchnorm_miopen_nhwc_compare_cpu

[dnikolaev-amd]

test rewritten to use native nhwc batchnorm to verify miopen non static PYTORCH_MIOPEN_SUGGEST_NHWC variable to be able to change it in runtime @jithunnair-amd @jeffdaily

[dnikolaev-amd]

Demo of enabling and disabling nhwc batchnorm on MIOpen in runtime by modifying PYTORCH_MIOPEN_SUGGEST_NHWC environment variable in runtime. Rename attached file to batchnorm_nhwc_enable_miopen_in_runtime.py and run it python -u batchnorm_nhwc_enable_miopen_in_runtime.py MIOpen(HIP): Command [LogCmdBNorm] ./bin/MIOpenDriver bnorm ... string can be observed while environment variable is enabled in python code batchnorm_nhwc_enable_miopen_in_runtime.py.txt

output

START
input.shape=torch.Size([20, 100, 5, 4])
input.stride=(2000, 1, 400, 100)
Disable MIOpen
PYTORCH_MIOPEN_SUGGEST_NHWC=None. No 'MIOpen(HIP):' output expected

Enable MIOpen
PYTORCH_MIOPEN_SUGGEST_NHWC=1. Expected 'MIOpen(HIP):' output
MIOpen(HIP): Command [LogCmdBNorm] ./bin/MIOpenDriver bnorm -n 20 -c 100 -H 5 -W 4 -M 1 --forw 1 -b 0 -s 1 -r 1

Disable MIOpen
PYTORCH_MIOPEN_SUGGEST_NHWC=0. No 'MIOpen(HIP):' output expected

output.shape=torch.Size([20, 100, 5, 4])
output.stride=(2000, 1, 400, 100)
END

[dnikolaev-amd]

test_nn.py enables to use MIOpen for NHWC batchnorms for whole file by defining PYTORCH_MIOPEN_SUGGEST_NHWC=1 on startup. test_batchnorm_nhwc_cuda used for demo as simplest nhwc batchnorm test. MIOPEN_ENABLE_LOGGING_CMD=1 enables miopen logging. String MIOpen(HIP): Command [LogCmdBNorm] ./bin/MIOpenDriver bnorm ... can be observed. So Pytorch used miopen for nhwc batchnorm without definition of PYTORCH_MIOPEN_SUGGEST_NHWC=1 from command line.

/var/lib/jenkins/pytorch# CI=1 PYTORCH_TEST_WITH_ROCM=1 MIOPEN_ENABLE_LOGGING_CMD=1 python test/run_test.py -v -i test_nn -k test_batchnorm_nhwc_cuda 

/var/lib/jenkins/pytorch/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
::warning:: Gathered no stats from artifacts for build env linux-focal-rocm6.0-py3.8 build env and None test config. Using default build env and default test config instead.
::warning:: Gathered no stats from artifacts for build env linux-focal-rocm6.0-py3.8 build env and None test config. Using default build env and default test config instead.
Running all tests
Running parallel tests on 2 processes
Name: tests to run (est. time: 3.55min)
  Serial tests (1):
    test_nn 1/1
  Parallel tests (0):
Name: excluded (est. time: 0.0min)
  Serial tests (0):
  Parallel tests (0):
Starting test batch 'tests to run' 0.0 seconds after initiating testing
No TD results found
Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json
Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
Running test_nn 1/1 ... [2024-06-04 15:54:43.585762]
Executing ['/opt/conda/envs/py_3.9/bin/python', '-bb', 'test_nn.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_batchnorm_nhwc_cuda', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-04 15:54:43.586660]

test_nn 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_nn_1.1_ac755ba8f1aea1a9_.log
Running 1 items in this shard: test/test_nn.py::TestNN::test_batchnorm_nhwc_cuda

 START of temporary dumping of test_nn 1/1 execution log file from (/var/lib/jenkins/pytorch/test/test-reports/test_nn_1.1_mr9hiz5z_toprint.log)
Test results will be stored in test-reports/python-pytest/test_nn/test_nn-1c4656d3af11c97e.xml
============================= test session starts ==============================
platform linux -- Python 3.9.19, pytest-7.3.2, pluggy-1.5.0 -- /opt/conda/envs/py_3.9/bin/python
cachedir: .pytest_cache
hypothesis profile 'pytorch_ci' -> database=None, deadline=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
rootdir: /var/lib/jenkins/pytorch
configfile: pytest.ini
plugins: flakefinder-1.1.0, cpp-2.3.0, xdist-3.3.1, hypothesis-5.35.1, xdoctest-1.1.0, rerunfailures-14.0
collecting ... collected 2193 items / 2192 deselected / 1 selected
stepcurrent: Cannot find last run test, not skipping
Running 1 items in this shard: test/test_nn.py::TestNN::test_batchnorm_nhwc_cuda

test_nn.py::TestNN::test_batchnorm_nhwc_cuda MIOpen(HIP): Command [LogCmdBNorm] ./bin/MIOpenDriver bnorm -n 2 -c 64 -H 50 -W 50 -M 1 --forw 2 -b 0 -s 1
MIOpen(HIP): Command [LogCmdBNorm] ./bin/MIOpenDriver bnorm -n 2 -c 64 -H 50 -W 50 -M 1 --forw 2 -b 0 -s 1
PASSED [1.4000s]            [100%]

- generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_nn/test_nn-1c4656d3af11c97e.xml -
====================== 1 passed, 2192 deselected in 1.59s ======================
END of temporary dumping of test_nn 1/1 execution log file form (/var/lib/jenkins/pytorch/test/test-reports/test_nn_1.1_mr9hiz5z_toprint.log)