hcc icon indicating copy to clipboard operation
hcc copied to clipboard

HCC_RUNTIME=CPU not working for rocm/tensorflow

Open ghostplant opened this issue 6 years ago • 6 comments

I force the runtime to use CPU mode but tensorflow is not able to launch the computation:

localhost# docker run -it --rm -e HIP_PLATFORM=hcc -e HCC_RUNTIME=CPU -v /:/externals rocm/tensorflow:rocm1.9.0-tf1.10-python3 bash

root@b9ba2c1cbb0e:/root# /externals/tf_examples/tf_matmul.py

2019-01-10 17:10:37.190768: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F
terminate called after throwing an instance of 'ihipException'
  what():  std::exception
Aborted (core dumped)

root@b9ba2c1cbb0e:/root#

ghostplant avatar Jan 10 '19 17:01 ghostplant

@ghostplant May I understand your desired goal? Do you wish to run this container rocm/tensorflow:rocm1.9.0-tf1.10-python3 and run all the tensorflow operators on CPU? If that's true, could you try this envrionment variable?

HIP_VISIBLE_DEVICES=-1 /externals/tf_examples/tf_matmul.py

HCC_RUNTIME=CPU was served for other purposes. To tune what GPUs are visible for HIP applications (such as TensorFlow), please use HIP_VISIBLE_DEVICES.

whchung avatar Jan 10 '19 17:01 whchung

Hi @whchung , HIP_VISIBLE_DEVICES=-1 is not my purpose. I want to test how hip data path uses libmcwamp_cpu.so when AMD GPU (or libmcwamp_hsa.so) is not available, because I think hcc was designed for both HSA-based platform and CPU-based platform, right? Such as Fallback to CPU when HSA fails, etc. So does this feature work for tensorflow over HIP?

ghostplant avatar Jan 10 '19 17:01 ghostplant

@ghostplant Unfortunately this route (forcing HCC runtime run on CPU and hope HIP applications work) is not supported.

whchung avatar Jan 10 '19 17:01 whchung

@whchung So do you mean libmcwamp_cpu.so is actually not useful?

ghostplant avatar Jan 10 '19 17:01 ghostplant

No, libmcwamp_cpu.so is not useful here. Unfortunately HCC_RUNTIME=CPU wasn't designed to let HIP applications run on CPU.

whchung avatar Jan 10 '19 17:01 whchung

@whchung Is there a user example to show what CPU-mode (libmcwamp_cpu.so) is used for? Thanks!

ghostplant avatar Jan 28 '19 15:01 ghostplant