java icon indicating copy to clipboard operation
java copied to clipboard

Published 20 hours ago • verified publisher icon tensorflow

Can't use GPU on latest snapshot

Open rnett opened this issue 4 years ago • 6 comments
trafficstars

On Windows, I can't use a GPU device even when depending on tensorflow-core-platform-gpu, and I don't see the dll finding output. Things work fine on 0.3.3, but not on 0.4.0-SNAPSHOT so I suspect it's related to the 2.6.0 upgrade.

rnett avatar Sep 19 '21 00:09 rnett

Set the "org.bytedeco.javacpp.logger.debug" system property to "true" to get additional information printed on the console.

saudet avatar Sep 19 '21 03:09 saudet

Here's the test:

import org.tensorflow.DeviceSpec
import org.tensorflow.op.Ops

fun main() {
    val tf = Ops.create().withDevice(DeviceSpec.newBuilder().deviceType(DeviceSpec.DeviceType.GPU).build())

    println(tf.constant(10).asTensor().getInt())
}

here's the log:

Debug: Loading class org.bytedeco.javacpp.presets.javacpp
Debug: Loading class org.bytedeco.javacpp.Loader
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-locale-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-string-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-stdio-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-math-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-heap-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-runtime-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-convert-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-environment-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-time-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-filesystem-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-utility-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-multibyte-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-conio-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-private-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-crt-process-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-console-l1-2-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-string-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-errorhandling-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-timezone-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-file-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-namedpipe-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-handle-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-file-l2-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-heap-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-libraryloader-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-synch-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-processthreads-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-processenvironment-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-datetime-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-localization-l1-2-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-sysinfo-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-synch-l1-2-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-console-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-debug-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-rtlsupport-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-processthreads-l1-1-1.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-file-l1-2-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-profile-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-memory-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-util-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\api-ms-win-core-interlocked-l1-1-0.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\ucrtbase.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\vcruntime140.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\vcruntime140_1.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\msvcp140.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\msvcp140_1.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\concrt140.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\vcomp140.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\javacpp-1.5.6-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\jnijavacpp.dll
Debug: Loading class org.bytedeco.javacpp.presets.javacpp
Debug: Loading class org.bytedeco.javacpp.Pointer
Debug: Loading class org.tensorflow.internal.c_api.global.tensorflow
Debug: Loading class org.tensorflow.internal.c_api.global.tensorflow
Debug: Loading library msvcr120
Debug: Loading C:\Users\jimne\.javacpp\cache\tensorflow-core-api-0.4.0-SNAPSHOT-windows-x86_64-gpu.jar\org\tensorflow\internal\c_api\windows-x86_64-gpu\libiomp5md.dll
Debug: Loading library mklml
Debug: Failed to load for mklml: java.lang.UnsatisfiedLinkError: no mklml in java.library.path: C:\Users\jimne\.jdks\openjdk-15.0.2\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\libnvvp;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\libnvvp;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Program Files\Microsoft MPI\Bin\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\dotnet\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\MySQL\MySQL Utilities 1.6\;C:\Program Files\wkhtmltopdf\bin;E:\Program Files (x86)\pgmodeler;C:\Program Files\7-Zip;C:\Program Files\Microsoft VS Code\bin;C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\bin;C:\Program Files\CMake\bin;C:\Program Files\LLVM\bin;C:\cudnn\v7\bin;C:\cudnn\v7;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\apache-maven-3.6.0\bin;C:\Program Files\swigwin;C:\Program Files\ffmpeg\bin;C:\Program Files\NVIDIA Corporation\NVSMI;C:\Scripts;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Go\bin;C:\Program Files (x86)\Gpg4win\..\GnuPG\bin;C:\Users\jimne\Desktop\OtherStuff\Bazelisk;C:\Program Files\visualvm_204\bin;C:\ProgramData\chocolatey\bin;C:\Program Files (x86)\Graphviz2.38\bin;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\PuTTY\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\cudnn\v8;C:\cudnn\v8\bin;C:\Program Files\nodejs\;C:\Users\jimne\.jdks\openjdk-15.0.2;C:\Users\jimne\.jdks\openjdk-15.0.2\bin;C:\Program Files (x86)\GnuWin32\bin;C:\Program Files\Git\cmd;C:\Program Files\Jetbrains CDN Layout\bin;C:\Program Files\NVIDIA Corporation\Nsight Compute 2020.3.0\;C:\Users\jimne\miniconda3;C:\Users\jimne\miniconda3\Library\mingw-w64\bin;C:\Users\jimne\miniconda3\Library\usr\bin;C:\Users\jimne\miniconda3\Library\bin;C:\Users\jimne\miniconda3\Scripts;C:\Users\jimne\.cargo\bin;C:\Users\jimne\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Microsoft VS Code\bin;C:\Users\jimne\AppData\Local\GitHubDesktop\bin;C:\Users\jimne\AppData\Local\Microsoft\WindowsApps;C:\Users\jimne\go\bin;C:\Program Files (x86)\Git Credential Manager Core;C:\Users\jimne\AppData\Roaming\npm;;.
Debug: Loading library tensorflow_framework
Debug: Failed to load for tensorflow_framework: java.lang.UnsatisfiedLinkError: no tensorflow_framework in java.library.path: C:\Users\jimne\.jdks\openjdk-15.0.2\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\libnvvp;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\libnvvp;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Program Files\Microsoft MPI\Bin\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\dotnet\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\MySQL\MySQL Utilities 1.6\;C:\Program Files\wkhtmltopdf\bin;E:\Program Files (x86)\pgmodeler;C:\Program Files\7-Zip;C:\Program Files\Microsoft VS Code\bin;C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\bin;C:\Program Files\CMake\bin;C:\Program Files\LLVM\bin;C:\cudnn\v7\bin;C:\cudnn\v7;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\apache-maven-3.6.0\bin;C:\Program Files\swigwin;C:\Program Files\ffmpeg\bin;C:\Program Files\NVIDIA Corporation\NVSMI;C:\Scripts;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Go\bin;C:\Program Files (x86)\Gpg4win\..\GnuPG\bin;C:\Users\jimne\Desktop\OtherStuff\Bazelisk;C:\Program Files\visualvm_204\bin;C:\ProgramData\chocolatey\bin;C:\Program Files (x86)\Graphviz2.38\bin;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\PuTTY\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\cudnn\v8;C:\cudnn\v8\bin;C:\Program Files\nodejs\;C:\Users\jimne\.jdks\openjdk-15.0.2;C:\Users\jimne\.jdks\openjdk-15.0.2\bin;C:\Program Files (x86)\GnuWin32\bin;C:\Program Files\Git\cmd;C:\Program Files\Jetbrains CDN Layout\bin;C:\Program Files\NVIDIA Corporation\Nsight Compute 2020.3.0\;C:\Users\jimne\miniconda3;C:\Users\jimne\miniconda3\Library\mingw-w64\bin;C:\Users\jimne\miniconda3\Library\usr\bin;C:\Users\jimne\miniconda3\Library\bin;C:\Users\jimne\miniconda3\Scripts;C:\Users\jimne\.cargo\bin;C:\Users\jimne\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Microsoft VS Code\bin;C:\Users\jimne\AppData\Local\GitHubDesktop\bin;C:\Users\jimne\AppData\Local\Microsoft\WindowsApps;C:\Users\jimne\go\bin;C:\Program Files (x86)\Git Credential Manager Core;C:\Users\jimne\AppData\Roaming\npm;;.
Debug: Loading C:\Users\jimne\.javacpp\cache\tensorflow-core-api-0.4.0-SNAPSHOT-windows-x86_64-gpu.jar\org\tensorflow\internal\c_api\windows-x86_64-gpu\tensorflow_cc.dll
Debug: Loading C:\Users\jimne\.javacpp\cache\tensorflow-core-api-0.4.0-SNAPSHOT-windows-x86_64-gpu.jar\org\tensorflow\internal\c_api\windows-x86_64-gpu\jnitensorflow.dll
Debug: Loading class org.bytedeco.javacpp.presets.javacpp
Debug: Loading class org.bytedeco.javacpp.BytePointer
Debug: Opening org.bytedeco.javacpp.PointerScope@326de728
Debug: Registering org.tensorflow.internal.c_api.TFE_ContextOptions[address=0x1570a5dff30,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTFE_ContextOptions$DeleteDeallocator[address=0x1570a5dff30,position=0,limit=0,capacity=0,deallocator=null],count=0]]
Debug: Attaching org.tensorflow.internal.c_api.TFE_ContextOptions[address=0x1570a5dff30,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTFE_ContextOptions$DeleteDeallocator[address=0x1570a5dff30,position=0,limit=0,capacity=0,deallocator=null],count=0]] to org.bytedeco.javacpp.PointerScope@326de728
Debug: Registering org.tensorflow.internal.c_api.TF_Status[address=0x15709bf29b0,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x15709bf29b0,position=0,limit=0,capacity=0,deallocator=null],count=0]]
Debug: Attaching org.tensorflow.internal.c_api.TF_Status[address=0x15709bf29b0,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x15709bf29b0,position=0,limit=0,capacity=0,deallocator=null],count=0]] to org.bytedeco.javacpp.PointerScope@326de728
2021-09-18 20:51:55.307007: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Debug: Closing org.bytedeco.javacpp.PointerScope@326de728
Debug: Releasing org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x15709bf29b0,position=0,limit=0,capacity=0,deallocator=null],count=0]
Debug: Releasing org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTFE_ContextOptions$DeleteDeallocator[address=0x1570a5dff30,position=0,limit=0,capacity=0,deallocator=null],count=0]
Debug: Loading class org.tensorflow.internal.c_api.global.tensorflow
Debug: Loading class org.tensorflow.internal.c_api.Deallocator_Pointer_long_Pointer
Debug: Registering org.tensorflow.internal.c_api.AbstractTF_Tensor$1[address=0x1570b63c8e0,position=0,limit=1,capacity=1,deallocator=org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x1570b63c8e0,deallocatorAddress=0x7ff908062030]]
Debug: Registering org.tensorflow.internal.c_api.TF_Tensor[address=0x1570b5bc6a0,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Tensor$DeleteDeallocator[address=0x1570b5bc6a0,position=0,limit=0,capacity=0,deallocator=null],count=0]]
Debug: Opening org.bytedeco.javacpp.PointerScope@573f2bb1
Debug: Attaching org.tensorflow.internal.c_api.TF_Tensor[address=0x1570b5bc6a0,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Tensor$DeleteDeallocator[address=0x1570b5bc6a0,position=0,limit=0,capacity=0,deallocator=null],count=0]] to org.bytedeco.javacpp.PointerScope@573f2bb1
Debug: Extending org.bytedeco.javacpp.PointerScope@573f2bb1
Debug: Closing org.bytedeco.javacpp.PointerScope@573f2bb1
Debug: Opening org.bytedeco.javacpp.PointerScope@42e26948
Debug: Registering org.tensorflow.internal.c_api.TF_Status[address=0x1570b5bc6d0,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x1570b5bc6d0,position=0,limit=0,capacity=0,deallocator=null],count=0]]
Debug: Attaching org.tensorflow.internal.c_api.TF_Status[address=0x1570b5bc6d0,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x1570b5bc6d0,position=0,limit=0,capacity=0,deallocator=null],count=0]] to org.bytedeco.javacpp.PointerScope@42e26948
Debug: Registering org.tensorflow.internal.c_api.TFE_Op[address=0x1570b667270,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTFE_Op$DeleteDeallocator[address=0x1570b667270,position=0,limit=0,capacity=0,deallocator=null],count=0]]
Debug: Attaching org.tensorflow.internal.c_api.TFE_Op[address=0x1570b667270,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTFE_Op$DeleteDeallocator[address=0x1570b667270,position=0,limit=0,capacity=0,deallocator=null],count=0]] to org.bytedeco.javacpp.PointerScope@42e26948
Debug: Closing org.bytedeco.javacpp.PointerScope@42e26948
Debug: Releasing org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x1570b5bc6d0,position=0,limit=0,capacity=0,deallocator=null],count=0]
Debug: Opening org.bytedeco.javacpp.PointerScope@1a968a59
Debug: Registering org.tensorflow.internal.c_api.TF_Status[address=0x1570b5bc590,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x1570b5bc590,position=0,limit=0,capacity=0,deallocator=null],count=0]]
Debug: Attaching org.tensorflow.internal.c_api.TF_Status[address=0x1570b5bc590,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x1570b5bc590,position=0,limit=0,capacity=0,deallocator=null],count=0]] to org.bytedeco.javacpp.PointerScope@1a968a59
Debug: Closing org.bytedeco.javacpp.PointerScope@1a968a59
Debug: Releasing org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x1570b5bc590,position=0,limit=0,capacity=0,deallocator=null],count=0]
Debug: Opening org.bytedeco.javacpp.PointerScope@204f30ec
Debug: Registering org.tensorflow.internal.c_api.TF_Status[address=0x1570b5bc6d0,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x1570b5bc6d0,position=0,limit=0,capacity=0,deallocator=null],count=0]]
Debug: Attaching org.tensorflow.internal.c_api.TF_Status[address=0x1570b5bc6d0,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x1570b5bc6d0,position=0,limit=0,capacity=0,deallocator=null],count=0]] to org.bytedeco.javacpp.PointerScope@204f30ec
Debug: Closing org.bytedeco.javacpp.PointerScope@204f30ec
Debug: Releasing org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x1570b5bc6d0,position=0,limit=0,capacity=0,deallocator=null],count=0]
Debug: Opening org.bytedeco.javacpp.PointerScope@e25b2fe
Debug: Loading class org.bytedeco.javacpp.presets.javacpp
Debug: Loading class org.bytedeco.javacpp.IntPointer
Debug: Registering org.bytedeco.javacpp.IntPointer[address=0x1570b5bc650,position=0,limit=1,capacity=1,deallocator=org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x1570b5bc650,deallocatorAddress=0x7ff9102f1b10]]
Debug: Attaching org.bytedeco.javacpp.IntPointer[address=0x1570b5bc650,position=0,limit=1,capacity=1,deallocator=org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x1570b5bc650,deallocatorAddress=0x7ff9102f1b10]] to org.bytedeco.javacpp.PointerScope@e25b2fe
Debug: Loading class org.bytedeco.javacpp.presets.javacpp
Debug: Loading class org.bytedeco.javacpp.PointerPointer
Debug: Registering org.bytedeco.javacpp.PointerPointer[address=0x1570b432f40,position=0,limit=1000,capacity=1000,deallocator=org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x1570b432f40,deallocatorAddress=0x7ff9102f1b10]]
Debug: Attaching org.bytedeco.javacpp.PointerPointer[address=0x1570b432f40,position=0,limit=1000,capacity=1000,deallocator=org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x1570b432f40,deallocatorAddress=0x7ff9102f1b10]] to org.bytedeco.javacpp.PointerScope@e25b2fe
Debug: Registering org.tensorflow.internal.c_api.TF_Status[address=0x1570b5bc610,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x1570b5bc610,position=0,limit=0,capacity=0,deallocator=null],count=0]]
Debug: Attaching org.tensorflow.internal.c_api.TF_Status[address=0x1570b5bc610,position=0,limit=0,capacity=0,deallocator=org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x1570b5bc610,position=0,limit=0,capacity=0,deallocator=null],count=0]] to org.bytedeco.javacpp.PointerScope@e25b2fe
Debug: Closing org.bytedeco.javacpp.PointerScope@e25b2fe
Debug: Releasing org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Status$DeleteDeallocator[address=0x1570b5bc610,position=0,limit=0,capacity=0,deallocator=null],count=0]
Debug: Releasing org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x1570b432f40,deallocatorAddress=0x7ff9102f1b10]
Debug: Collecting org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x0,deallocatorAddress=0x0]
Debug: Releasing org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x1570b5bc650,deallocatorAddress=0x7ff9102f1b10]
Debug: Collecting org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x0,deallocatorAddress=0x0]
Debug: Closing org.bytedeco.javacpp.PointerScope@573f2bb1
Debug: Releasing org.bytedeco.javacpp.Pointer$DeallocatorReference[deallocator=org.tensorflow.internal.c_api.AbstractTF_Tensor$DeleteDeallocator[address=0x1570b5bc6a0,position=0,limit=0,capacity=0,deallocator=null],count=0]
Exception in thread "main" org.tensorflow.exceptions.TFInvalidArgumentException: Could not satisfy device specification '/device:GPU:*'. enable_soft_placement=0. Supported device types [CPU]. All available devices [/job:localhost/replica:0/task:0/device:CPU:0].
	at org.tensorflow.internal.c_api.AbstractTF_Status.throwExceptionIfNotOK(AbstractTF_Status.java:87)
	at org.tensorflow.EagerOperationBuilder.execute(EagerOperationBuilder.java:306)
	at org.tensorflow.EagerOperationBuilder.build(EagerOperationBuilder.java:75)
	at org.tensorflow.EagerOperationBuilder.build(EagerOperationBuilder.java:62)
	at org.tensorflow.op.core.Constant.create(Constant.java:1350)
	at org.tensorflow.op.core.Constant.scalarOf(Constant.java:85)
	at org.tensorflow.op.Ops.constant(Ops.java:1292)
	at tester.GPUTestKt.main(GPUTest.kt:9)
	at tester.GPUTestKt.main(GPUTest.kt)

The only suspicous thing I see is:

Debug: Loading library tensorflow_framework
Debug: Failed to load for tensorflow_framework: java.lang.UnsatisfiedLinkError: no tensorflow_framework in java.library.path

and I see that on 0.3.3 as well (where it works fine).

rnett avatar Sep 19 '21 03:09 rnett

Comparing out .bazelrc to tensorflow's, it seems like the cuda options changed, i.e. there is no using_cuda any more. Additionally, our .bazelrc refers to cuda 10.1 while tensorflow's uses 11.2. That's my suspicion for the culprit.

rnett avatar Sep 19 '21 03:09 rnett

We really should try to get some GPU CI tests.

rnett avatar Sep 19 '21 03:09 rnett

We really should try to get some GPU CI tests.

+1 to that, but it's been a while since I heard from Google folks even after pinging them many times for this specific request.

Let's update the bazelrc file and see how it goes then, I was also suspecting that we needed to merge it in ours. Are you able to test that out locally @rnett or should we push right away a new version and see how it goes?

karllessard avatar Sep 19 '21 16:09 karllessard

I should be able to test it locally, I'm running native builds for the custom gradient PR anyways.

I'm also seeing a new op QuantizedConcatV2 when I build locally, that doesn't seem to have been added in the 2.6 update.

rnett avatar Sep 19 '21 22:09 rnett