tensorflow-opencl icon indicating copy to clipboard operation
tensorflow-opencl copied to clipboard

Getting Nan everywhere

Open SavvaI opened this issue 7 years ago • 4 comments

Hello, I literally followed this guide from end to end https://www.codeplay.com/portal/03-30-17-setting-up-tensorflow-with-opencl-using-sycl , and it kinda worked, (i tested multiplying small matrices) but when i run any of tensorflow examples ( https://github.com/aymericdamien/TensorFlow-Examples ) i am getting Nan in training loss everywhere, also batch computation time is very long. I have AMD Radeon R9 290, ubuntu 14.04

SavvaI avatar Jul 01 '17 08:07 SavvaI

The latest guide docs are here https://www.codeplay.com/products/computesuite/computecpp/guides/how-to-setup-tensorflow-with-computecpp and you would be better trying the branch at https://github.com/lukeiwanski/tensorflow.git which has the latest code base.

rodburns avatar Jul 03 '17 09:07 rodburns

I tried that guide and it didn't help. Executing mnist example from the guide works, but i get 263ms batch time comparing with 157ms when working on CPU

SavvaI avatar Jul 06 '17 11:07 SavvaI

That is not unexpected at the moment, we are adding vectorization which optimizes the implementation. This is being prepared as a pull request and will be available soon.

rodburns avatar Jul 06 '17 14:07 rodburns

Ok. Thank you for reply.

SavvaI avatar Jul 07 '17 05:07 SavvaI