Running tests on XLA:GPU takes forever

[alanwaketan]

It takes forever to run any tests on XLA GPU. And suspicious messages are shown:

(pytorch) jwtan@jwtan-v100-4:~/work/pytorch/xla$ MASTER_ADDR=localhost MASTER_PORT=6000 LD_LIBRARY_PATH=/opt/conda/lib/ python test/test_ddp.py TestXrtDistributedDataParallel.test_ddp_correctness
Running tests under Python 3.10.6: /opt/conda/envs/pytorch/bin/python3
[ RUN      ] TestXrtDistributedDataParallel.test_ddp_correctness
2022-10-12 07:41:58.001969: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2027] TensorFlow was not built with CUDA kernel binaries compatible with compute capability 7.0. CUDA kernels will be jit-compiled from PTX, which could take 30 minutes or longer.
2022-10-12 07:41:58.025166: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2027] TensorFlow was not built with CUDA kernel binaries compatible with compute capability 7.0. CUDA kernels will be jit-compiled from PTX, which could take 30 minutes or longer.
2022-10-12 07:41:58.058488: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2027] TensorFlow was not built with CUDA kernel binaries compatible with compute capability 7.0. CUDA kernels will be jit-compiled from PTX, which could take 30 minutes or longer.
2022-10-12 07:41:58.155964: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2027] TensorFlow was not built with CUDA kernel binaries compatible with compute capability 7.0. CUDA kernels will be jit-compiled from PTX, which could take 30 minutes or longer.