rnnt-speech-recognition
rnnt-speech-recognition copied to clipboard
Published
20 hours ago •
noahchalifour
noahchalifour
Getting ValueError: Attempt to convert a value (PerReplica ..) when starting training
Hi!
I am currently trying to start a simple training by following the instructions from the README.md. Everything works up to the point where I want to start the training.
Executing
python run_rnnt.py --mode train --data_dir /home/sfalk/pt/shards/
Throws
ValueError: Attempt to convert a value (PerReplica ..)
Click to expand full error log
/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/librosa/util/decorators.py:9: NumbaDeprecationWarning: An import was requested from a module that has moved location.
Import requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0.
from numba.decorators import jit as optional_jit
/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/librosa/util/decorators.py:9: NumbaDeprecationWarning: An import was requested from a module that has moved location.
Import of 'jit' requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0.
from numba.decorators import jit as optional_jit
2020-05-26 09:14:30.736191: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-26 09:14:30.748386: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.749173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:30.749232: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.750058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 1 with properties:
pciBusID: 0000:02:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:30.750112: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.750888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 2 with properties:
pciBusID: 0000:03:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:30.750927: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.751427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 3 with properties:
pciBusID: 0000:05:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:30.751570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-26 09:14:30.752638: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-26 09:14:30.753673: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-26 09:14:30.753866: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-26 09:14:30.754997: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-26 09:14:30.755618: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-26 09:14:30.757804: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-26 09:14:30.757899: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.759345: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.760097: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.760844: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.761589: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.762328: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.763068: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.763805: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.764521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0, 1, 2, 3
2020-05-26 09:14:30.764770: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-05-26 09:14:30.770070: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 4200000000 Hz
2020-05-26 09:14:30.770492: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560e6150fef0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-26 09:14:30.770507: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-05-26 09:14:31.020514: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.037961: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.041811: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.049635: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.050189: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560e60e72d20 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-26 09:14:31.050199: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-05-26 09:14:31.050203: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (1): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-05-26 09:14:31.050206: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (2): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-05-26 09:14:31.050209: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (3): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-05-26 09:14:31.051527: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.051949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:31.051989: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.052409: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 1 with properties:
pciBusID: 0000:02:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:31.052448: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.052867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 2 with properties:
pciBusID: 0000:03:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:31.052904: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.053326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 3 with properties:
pciBusID: 0000:05:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:31.053353: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-26 09:14:31.053366: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-26 09:14:31.053377: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-26 09:14:31.053387: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-26 09:14:31.053397: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-26 09:14:31.053407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-26 09:14:31.053418: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-26 09:14:31.053452: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.053895: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.054339: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.054782: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.055227: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.055669: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.056126: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.056579: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.057003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0, 1, 2, 3
2020-05-26 09:14:31.057025: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-26 09:14:31.059325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-26 09:14:31.059335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 1 2 3
2020-05-26 09:14:31.059340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N Y Y Y
2020-05-26 09:14:31.059344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 1: Y N Y Y
2020-05-26 09:14:31.059347: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 2: Y Y N Y
2020-05-26 09:14:31.059350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 3: Y Y Y N
2020-05-26 09:14:31.060102: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.060567: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.061033: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.061486: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.061942: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.062368: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9449 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-05-26 09:14:31.062688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.063131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10161 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2020-05-26 09:14:31.063473: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.064690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10161 MB memory) -> physical GPU (device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1)
2020-05-26 09:14:31.065011: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.065455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10161 MB memory) -> physical GPU (device: 3, name: GeForce GTX 1080 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
4 Physical GPU, 4 Logical GPUs
WARNING:tensorflow:From /home/sfalk/tmp/rnnt-speech-recognition/model.py:59: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
W0526 09:14:32.108052 140106746382080 deprecation.py:317] From /home/sfalk/tmp/rnnt-speech-recognition/model.py:59: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c9e97d820>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.108385 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c9e97d820>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:From /home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/ops/rnn_cell_impl.py:962: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.
W0526 09:14:32.109819 140106746382080 deprecation.py:317] From /home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/ops/rnn_cell_impl.py:962: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c9009e730>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.227335 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c9009e730>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c4811f9d0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.490125 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c4811f9d0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c48086070>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.669947 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c48086070>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c383bc4f0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.804272 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c383bc4f0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c383a7d00>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.951039 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c383a7d00>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c3830e190>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:33.074690 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c3830e190>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c382f8250>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:33.202479 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c382f8250>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c381bd820>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:33.890956 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c381bd820>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c086b71f0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:34.015121 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c086b71f0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
I0526 09:14:34.344151 140106746382080 run_rnnt.py:490] Using word-piece encoder with vocab size: 4341
Model: "encoder"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, None, 240)] 0
_________________________________________________________________
batch_normalization (BatchNo (None, None, 240) 960
_________________________________________________________________
rnn (RNN) (None, None, 640) 8527872
_________________________________________________________________
dropout (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization (LayerNo (None, None, 640) 1280
_________________________________________________________________
rnn_1 (RNN) (None, None, 640) 11804672
_________________________________________________________________
dropout_1 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_1 (Layer (None, None, 640) 1280
_________________________________________________________________
time_reduction (TimeReductio (None, None, 1280) 0
_________________________________________________________________
rnn_2 (RNN) (None, None, 640) 17047552
_________________________________________________________________
dropout_2 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_2 (Layer (None, None, 640) 1280
_________________________________________________________________
rnn_3 (RNN) (None, None, 640) 11804672
_________________________________________________________________
dropout_3 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_3 (Layer (None, None, 640) 1280
_________________________________________________________________
rnn_4 (RNN) (None, None, 640) 11804672
_________________________________________________________________
dropout_4 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_4 (Layer (None, None, 640) 1280
_________________________________________________________________
rnn_5 (RNN) (None, None, 640) 11804672
_________________________________________________________________
dropout_5 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_5 (Layer (None, None, 640) 1280
_________________________________________________________________
rnn_6 (RNN) (None, None, 640) 11804672
_________________________________________________________________
dropout_6 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_6 (Layer (None, None, 640) 1280
_________________________________________________________________
rnn_7 (RNN) (None, None, 640) 11804672
_________________________________________________________________
dropout_7 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_7 (Layer (None, None, 640) 1280
=================================================================
Total params: 96,414,656
Trainable params: 96,414,176
Non-trainable params: 480
_________________________________________________________________
Model: "prediction_network"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_2 (InputLayer) [(None, None)] 0
_________________________________________________________________
embedding (Embedding) (None, None, 500) 2170500
_________________________________________________________________
rnn_8 (RNN) (None, None, 640) 10657792
_________________________________________________________________
dropout_8 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_8 (Layer (None, None, 640) 1280
_________________________________________________________________
rnn_9 (RNN) (None, None, 640) 11804672
_________________________________________________________________
dropout_9 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_9 (Layer (None, None, 640) 1280
=================================================================
Total params: 24,635,524
Trainable params: 24,635,524
Non-trainable params: 0
_________________________________________________________________
Model: "transducer"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
mel_specs (InputLayer) [(None, None, 240)] 0
__________________________________________________________________________________________________
pred_inp (InputLayer) [(None, None)] 0
__________________________________________________________________________________________________
encoder (Model) (None, None, 640) 96414656 mel_specs[0][0]
__________________________________________________________________________________________________
prediction_network (Model) (None, None, 640) 24635524 pred_inp[0][0]
__________________________________________________________________________________________________
tf_op_layer_ExpandDims (TensorF [(None, None, 1, 640 0 encoder[1][0]
__________________________________________________________________________________________________
tf_op_layer_ExpandDims_1 (Tenso [(None, 1, None, 640 0 prediction_network[1][0]
__________________________________________________________________________________________________
tf_op_layer_AddV2 (TensorFlowOp [(None, None, None, 0 tf_op_layer_ExpandDims[0][0]
tf_op_layer_ExpandDims_1[0][0]
__________________________________________________________________________________________________
dense (Dense) (None, None, None, 6 410240 tf_op_layer_AddV2[0][0]
__________________________________________________________________________________________________
dense_1 (Dense) (None, None, None, 4 2782581 dense[0][0]
==================================================================================================
Total params: 124,243,001
Trainable params: 124,242,521
Non-trainable params: 480
__________________________________________________________________________________________________
Starting training.
Performing evaluation.
Traceback (most recent call last):
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2292, in _convert_inputs_to_signature
flatten_inputs[index] = ops.convert_to_tensor(
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1341, in convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 321, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 261, in constant
return _constant_impl(value, dtype, shape, name, verify_shape=False,
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 270, in _constant_impl
t = convert_to_eager_tensor(value, ctx, dtype)
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 96, in convert_to_eager_tensor
return ops.EagerTensor(value, ctx.device_name, dtype)
ValueError: Attempt to convert a value (PerReplica:{
0: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-9.8887777e+00, -9.5391264e+00, -9.2146311e+00, ...,
1.4807711e+00, 1.4137149e+00, 1.5833356e+00],
[-2.5297828e+00, -1.0314496e+00, -4.4551528e-01, ...,
-8.5550594e-01, -3.8671780e-01, -6.2595654e-01],
[-9.2890608e-01, -9.3925929e-01, -1.0737282e+00, ...,
-4.6040058e-01, -1.3226795e-01, -4.6705770e-01],
...,
[-8.9524627e-02, -1.3095784e-01, 4.4763446e-02, ...,
-7.6179504e-03, 3.1356859e-01, 1.3805485e-01],
[-1.0855615e-01, -3.6668968e-01, -3.5269606e-01, ...,
-1.6952515e-01, -4.3339968e-01, -2.3297167e-01],
[-3.4607446e-01, -4.6576285e-01, -2.4114418e-01, ...,
-3.9931583e-01, -6.5470409e-01, -5.9117317e-02]],
[[-1.0188073e+01, -9.7674351e+00, -9.2495003e+00, ...,
2.7845190e+00, 3.0497322e+00, 2.9723659e+00],
[-5.8545446e-01, -9.0612173e-01, -1.4650891e+00, ...,
-1.2318707e+00, -1.2984281e+00, -1.2217040e+00],
[-4.9245834e-01, -6.1498523e-01, -9.3534470e-01, ...,
-1.1765385e+00, -1.4464822e+00, -6.3945484e-01],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
[[-9.7931051e+00, -9.2576466e+00, -8.7187033e+00, ...,
2.6441813e+00, 2.5935564e+00, 2.7019000e+00],
[-1.1656429e+00, -2.4396741e-01, 2.2748601e-01, ...,
-1.1423826e+00, -1.9891844e+00, -1.8549285e+00],
[-1.1924456e+00, -1.5766211e+00, -1.9995271e+00, ...,
-6.9014835e-01, -1.1843119e+00, -1.7883348e+00],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
...,
[[-9.2142944e+00, -8.7442112e+00, -8.0906887e+00, ...,
2.5637565e+00, 2.6928663e+00, 2.9714472e+00],
[-1.4353551e+00, -1.3912488e+00, -1.1410775e+00, ...,
-6.8289948e-01, -7.4661875e-01, -9.4346857e-01],
[-1.3514730e+00, -1.3447710e+00, -1.1697345e+00, ...,
-8.4204197e-01, -6.4308357e-01, -6.6116714e-01],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
[[-9.4380770e+00, -9.2610064e+00, -9.3578424e+00, ...,
1.3151474e+00, 1.3194189e+00, 1.6185846e+00],
[-1.4748455e+00, -9.6358955e-01, -7.5807011e-01, ...,
-1.2112403e+00, -5.9023190e-01, -2.0361996e-01],
[-1.2093054e+00, -1.8930241e+00, -1.3843757e+00, ...,
-3.7091255e-01, -6.0484600e-01, -1.9562192e+00],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
[[-9.2918978e+00, -9.2315750e+00, -8.9223146e+00, ...,
2.7278879e+00, 2.8609688e+00, 2.7423246e+00],
[-5.9187198e-01, -1.8667960e-01, 2.1728778e-01, ...,
-9.7960901e-01, -2.1147966e-01, -8.8720083e-01],
[-8.6280262e-01, -8.0901504e-01, -5.0346565e-01, ...,
-7.1112490e-01, -1.2980080e-01, 3.8867188e-01],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]]], dtype=float32)>,
1: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-10.0592985 , -9.730883 , -9.271258 , ..., 2.8219585 ,
2.6701574 , 3.145619 ],
[ -0.9512273 , -0.57793355, -0.37342393, ..., -0.36957645,
-0.58688927, -0.6034956 ],
[ -1.0401095 , -1.0510697 , -1.1356157 , ..., -0.2553835 ,
-0.40022898, -1.2278776 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.3389225 , -9.18263 , -8.9984045 , ..., 2.095945 ,
2.1119766 , 2.3430693 ],
[ -0.9921645 , -0.9324732 , -0.67674816, ..., -0.8709469 ,
-0.87235546, -0.8106818 ],
[ -2.0774326 , -2.127688 , -1.135358 , ..., -0.97692585,
-1.1275706 , -0.88771343],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[-10.804321 , -10.686319 , -10.408899 , ..., 2.3665676 ,
2.5379932 , 2.420307 ],
[ -1.1987064 , -1.067873 , -0.9529356 , ..., -0.30030394,
-0.12877178, -0.16108751],
[ -0.45661426, -0.7980306 , -1.1557912 , ..., -1.5581841 ,
-0.4684105 , -0.44000912],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
...,
[[ -9.445232 , -9.460578 , -9.285489 , ..., 1.8623145 ,
1.9304647 , 2.195097 ],
[ -1.0719146 , -1.0908513 , -0.9582559 , ..., -0.99088955,
-0.6627326 , -0.6685376 ],
[ -1.8296288 , -1.8624439 , -1.321793 , ..., -0.70586205,
-0.97963333, -1.346128 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.328621 , -9.586381 , -9.817234 , ..., 3.6245484 ,
3.8879015 , 3.8824553 ],
[ -2.2337832 , -1.1850247 , -0.43120062, ..., -1.3237286 ,
-0.8165655 , -0.5502229 ],
[ -4.9042144 , -1.1433594 , -0.3725183 , ..., 0.19899416,
0.02039671, 0.09885693],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.299686 , -8.799611 , -8.434992 , ..., 3.097272 ,
3.1601548 , 3.1777468 ],
[ -1.1876402 , -1.0250216 , -0.64193165, ..., -0.32103252,
-0.06696558, -0.6231766 ],
[ -0.5330059 , -0.6796452 , -0.77790976, ..., -0.9301033 ,
-0.8343253 , -0.95349455],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]]], dtype=float32)>,
2: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-10.20004 , -9.797416 , -9.144037 , ..., 2.1522975 ,
2.177644 , 2.4891276 ],
[ -0.6829766 , -1.0327927 , -1.9740211 , ..., -1.109499 ,
-1.2662077 , -1.4992175 ],
[ -0.50412035, -0.75838935, -1.4640208 , ..., -1.1935134 ,
-1.90482 , -1.4951177 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.057947 , -8.835877 , -8.432611 , ..., 2.7456422 ,
2.6064804 , 2.9408405 ],
[ -0.7960679 , -0.7536596 , -0.56650114, ..., -0.9725952 ,
-0.9330125 , -0.8314767 ],
[ -1.273035 , -1.0706124 , -0.8415742 , ..., -1.0601668 ,
-1.3329258 , -2.0243492 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -8.598597 , -8.078391 , -7.316219 , ..., 3.475574 ,
3.5632377 , 3.4421172 ],
[ -0.15228844, 0.27448 , 0.84499 , ..., -1.2552052 ,
-0.435256 , -0.70891285],
[ -0.08234251, -0.10229468, -0.20772624, ..., 0.49269104,
-0.33324862, -0.30197382],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
...,
[[-10.674739 , -10.312547 , -9.784449 , ..., 1.9142385 ,
2.0289752 , 2.207048 ],
[ -2.3002534 , -1.4031292 , -0.7912599 , ..., 0.46009445,
0.26866436, 0.23870277],
[ -0.8622447 , -0.8987092 , -0.7318269 , ..., -0.51101494,
0.02473354, 0.04263115],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[-11.913138 , -11.890651 , -11.718769 , ..., -0.6516266 ,
-0.11420345, -0.2889433 ],
[ -2.3279111 , -0.83327794, -0.38966238, ..., -0.18499136,
-0.31412745, -0.78526115],
[ -1.2105832 , -0.65170074, -0.30061066, ..., -0.5946727 ,
-0.5343251 , -0.26280165],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.108984 , -8.4880495 , -7.7428994 , ..., 3.0713623 ,
3.0152838 , 3.4003892 ],
[ -0.19017792, -0.29030776, -0.33207464, ..., -0.0172677 ,
-0.24603939, -0.9277673 ],
[ -0.6175966 , -0.8139999 , -1.1248744 , ..., -0.649323 ,
-0.6167655 , -0.5438595 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]]], dtype=float32)>,
3: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[ -8.988267 , -8.965929 , -8.728106 , ..., 2.1489723 ,
2.334958 , 2.5260453 ],
[ -0.9514227 , -0.8912209 , -0.42839706, ..., -0.7747984 ,
-0.88131857, -1.1858144 ],
[ -1.6392798 , -1.8752912 , -1.3590232 , ..., -0.71193075,
-0.46426392, -0.79004574],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.117433 , -8.756327 , -8.3021965 , ..., 2.6770406 ,
2.9165506 , 2.619464 ],
[ -0.8334371 , -0.83099353, -0.7541424 , ..., -1.08567 ,
-0.3741293 , -0.14404964],
[ -1.1120342 , -1.2024816 , -1.1394764 , ..., -1.3766332 ,
-1.0821438 , -0.808341 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -8.535436 , -8.446434 , -8.433824 , ..., 1.7634034 ,
1.9151568 , 2.234317 ],
[ -0.73761785, -1.0498412 , -1.1566993 , ..., -1.9087753 ,
-1.4135947 , -0.43615532],
[ -3.1867118 , -1.3442845 , -0.6116022 , ..., -1.2935882 ,
-1.5357218 , -1.4339061 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
...,
[[ -8.752668 , -8.795702 , -9.234339 , ..., 1.4543443 ,
1.308625 , 1.8872037 ],
[ -0.8426125 , -0.5590738 , -0.64102423, ..., -0.2699995 ,
-0.34920692, -0.23712015],
[ -0.49331522, -1.201171 , -1.1223062 , ..., -1.4179659 ,
-0.8049512 , -0.8301816 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.640016 , -9.385439 , -9.126853 , ..., 2.6549459 ,
2.8241823 , 3.1816008 ],
[ -1.9176644 , -1.4502705 , -0.9663389 , ..., -0.6905961 ,
-0.6539016 , -0.5099206 ],
[ -1.4909953 , -1.6704174 , -1.1156216 , ..., -1.4263935 ,
-0.81988144, -1.1464152 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[-11.273719 , -10.959808 , -10.497703 , ..., 1.7455099 ,
1.9854834 , 2.464494 ],
[ -2.0011683 , -1.4769925 , -0.9517286 , ..., -0.25176716,
-0.5848155 , -0.5404253 ],
[ -1.4961203 , -1.4866172 , -1.359714 , ..., -1.1029644 ,
-1.4994144 , -0.89810133],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]]], dtype=float32)>
}) with an unsupported type (<class 'tensorflow.python.distribute.values.PerReplica'>) to a Tensor.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "run_rnnt.py", line 586, in <module>
app.run(main)
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "run_rnnt.py", line 532, in main
run_training(
File "run_rnnt.py", line 347, in run_training
checkpoint_model()
File "run_rnnt.py", line 304, in checkpoint_model
eval_loss, eval_metrics_results = run_evaluate(
File "run_rnnt.py", line 433, in run_evaluate
loss, metrics_results = eval_step(inputs)
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
result = self._call(*args, **kwds)
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 647, in _call
self._stateful_fn._function_spec.canonicalize_function_inputs( # pylint: disable=protected-access
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2235, in canonicalize_function_inputs
inputs = _convert_inputs_to_signature(
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2296, in _convert_inputs_to_signature
raise ValueError("When input_signature is provided, all inputs to "
ValueError: When input_signature is provided, all inputs to the Python function must be convertible to tensors:
inputs: (
(PerReplica:{
0: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-9.8887777e+00, -9.5391264e+00, -9.2146311e+00, ...,
1.4807711e+00, 1.4137149e+00, 1.5833356e+00],
[-2.5297828e+00, -1.0314496e+00, -4.4551528e-01, ...,
-8.5550594e-01, -3.8671780e-01, -6.2595654e-01],
[-9.2890608e-01, -9.3925929e-01, -1.0737282e+00, ...,
-4.6040058e-01, -1.3226795e-01, -4.6705770e-01],
...,
[-8.9524627e-02, -1.3095784e-01, 4.4763446e-02, ...,
-7.6179504e-03, 3.1356859e-01, 1.3805485e-01],
[-1.0855615e-01, -3.6668968e-01, -3.5269606e-01, ...,
-1.6952515e-01, -4.3339968e-01, -2.3297167e-01],
[-3.4607446e-01, -4.6576285e-01, -2.4114418e-01, ...,
-3.9931583e-01, -6.5470409e-01, -5.9117317e-02]],
[[-1.0188073e+01, -9.7674351e+00, -9.2495003e+00, ...,
2.7845190e+00, 3.0497322e+00, 2.9723659e+00],
[-5.8545446e-01, -9.0612173e-01, -1.4650891e+00, ...,
-1.2318707e+00, -1.2984281e+00, -1.2217040e+00],
[-4.9245834e-01, -6.1498523e-01, -9.3534470e-01, ...,
-1.1765385e+00, -1.4464822e+00, -6.3945484e-01],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
[[-9.7931051e+00, -9.2576466e+00, -8.7187033e+00, ...,
2.6441813e+00, 2.5935564e+00, 2.7019000e+00],
[-1.1656429e+00, -2.4396741e-01, 2.2748601e-01, ...,
-1.1423826e+00, -1.9891844e+00, -1.8549285e+00],
[-1.1924456e+00, -1.5766211e+00, -1.9995271e+00, ...,
-6.9014835e-01, -1.1843119e+00, -1.7883348e+00],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
...,
[[-9.2142944e+00, -8.7442112e+00, -8.0906887e+00, ...,
2.5637565e+00, 2.6928663e+00, 2.9714472e+00],
[-1.4353551e+00, -1.3912488e+00, -1.1410775e+00, ...,
-6.8289948e-01, -7.4661875e-01, -9.4346857e-01],
[-1.3514730e+00, -1.3447710e+00, -1.1697345e+00, ...,
-8.4204197e-01, -6.4308357e-01, -6.6116714e-01],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
[[-9.4380770e+00, -9.2610064e+00, -9.3578424e+00, ...,
1.3151474e+00, 1.3194189e+00, 1.6185846e+00],
[-1.4748455e+00, -9.6358955e-01, -7.5807011e-01, ...,
-1.2112403e+00, -5.9023190e-01, -2.0361996e-01],
[-1.2093054e+00, -1.8930241e+00, -1.3843757e+00, ...,
-3.7091255e-01, -6.0484600e-01, -1.9562192e+00],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
[[-9.2918978e+00, -9.2315750e+00, -8.9223146e+00, ...,
2.7278879e+00, 2.8609688e+00, 2.7423246e+00],
[-5.9187198e-01, -1.8667960e-01, 2.1728778e-01, ...,
-9.7960901e-01, -2.1147966e-01, -8.8720083e-01],
[-8.6280262e-01, -8.0901504e-01, -5.0346565e-01, ...,
-7.1112490e-01, -1.2980080e-01, 3.8867188e-01],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]]], dtype=float32)>,
1: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-10.0592985 , -9.730883 , -9.271258 , ..., 2.8219585 ,
2.6701574 , 3.145619 ],
[ -0.9512273 , -0.57793355, -0.37342393, ..., -0.36957645,
-0.58688927, -0.6034956 ],
[ -1.0401095 , -1.0510697 , -1.1356157 , ..., -0.2553835 ,
-0.40022898, -1.2278776 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.3389225 , -9.18263 , -8.9984045 , ..., 2.095945 ,
2.1119766 , 2.3430693 ],
[ -0.9921645 , -0.9324732 , -0.67674816, ..., -0.8709469 ,
-0.87235546, -0.8106818 ],
[ -2.0774326 , -2.127688 , -1.135358 , ..., -0.97692585,
-1.1275706 , -0.88771343],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[-10.804321 , -10.686319 , -10.408899 , ..., 2.3665676 ,
2.5379932 , 2.420307 ],
[ -1.1987064 , -1.067873 , -0.9529356 , ..., -0.30030394,
-0.12877178, -0.16108751],
[ -0.45661426, -0.7980306 , -1.1557912 , ..., -1.5581841 ,
-0.4684105 , -0.44000912],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
...,
[[ -9.445232 , -9.460578 , -9.285489 , ..., 1.8623145 ,
1.9304647 , 2.195097 ],
[ -1.0719146 , -1.0908513 , -0.9582559 , ..., -0.99088955,
-0.6627326 , -0.6685376 ],
[ -1.8296288 , -1.8624439 , -1.321793 , ..., -0.70586205,
-0.97963333, -1.346128 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.328621 , -9.586381 , -9.817234 , ..., 3.6245484 ,
3.8879015 , 3.8824553 ],
[ -2.2337832 , -1.1850247 , -0.43120062, ..., -1.3237286 ,
-0.8165655 , -0.5502229 ],
[ -4.9042144 , -1.1433594 , -0.3725183 , ..., 0.19899416,
0.02039671, 0.09885693],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.299686 , -8.799611 , -8.434992 , ..., 3.097272 ,
3.1601548 , 3.1777468 ],
[ -1.1876402 , -1.0250216 , -0.64193165, ..., -0.32103252,
-0.06696558, -0.6231766 ],
[ -0.5330059 , -0.6796452 , -0.77790976, ..., -0.9301033 ,
-0.8343253 , -0.95349455],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]]], dtype=float32)>,
2: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-10.20004 , -9.797416 , -9.144037 , ..., 2.1522975 ,
2.177644 , 2.4891276 ],
[ -0.6829766 , -1.0327927 , -1.9740211 , ..., -1.109499 ,
-1.2662077 , -1.4992175 ],
[ -0.50412035, -0.75838935, -1.4640208 , ..., -1.1935134 ,
-1.90482 , -1.4951177 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.057947 , -8.835877 , -8.432611 , ..., 2.7456422 ,
2.6064804 , 2.9408405 ],
[ -0.7960679 , -0.7536596 , -0.56650114, ..., -0.9725952 ,
-0.9330125 , -0.8314767 ],
[ -1.273035 , -1.0706124 , -0.8415742 , ..., -1.0601668 ,
-1.3329258 , -2.0243492 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -8.598597 , -8.078391 , -7.316219 , ..., 3.475574 ,
3.5632377 , 3.4421172 ],
[ -0.15228844, 0.27448 , 0.84499 , ..., -1.2552052 ,
-0.435256 , -0.70891285],
[ -0.08234251, -0.10229468, -0.20772624, ..., 0.49269104,
-0.33324862, -0.30197382],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
...,
[[-10.674739 , -10.312547 , -9.784449 , ..., 1.9142385 ,
2.0289752 , 2.207048 ],
[ -2.3002534 , -1.4031292 , -0.7912599 , ..., 0.46009445,
0.26866436, 0.23870277],
[ -0.8622447 , -0.8987092 , -0.7318269 , ..., -0.51101494,
0.02473354, 0.04263115],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[-11.913138 , -11.890651 , -11.718769 , ..., -0.6516266 ,
-0.11420345, -0.2889433 ],
[ -2.3279111 , -0.83327794, -0.38966238, ..., -0.18499136,
-0.31412745, -0.78526115],
[ -1.2105832 , -0.65170074, -0.30061066, ..., -0.5946727 ,
-0.5343251 , -0.26280165],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.108984 , -8.4880495 , -7.7428994 , ..., 3.0713623 ,
3.0152838 , 3.4003892 ],
[ -0.19017792, -0.29030776, -0.33207464, ..., -0.0172677 ,
-0.24603939, -0.9277673 ],
[ -0.6175966 , -0.8139999 , -1.1248744 , ..., -0.649323 ,
-0.6167655 , -0.5438595 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]]], dtype=float32)>,
3: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[ -8.988267 , -8.965929 , -8.728106 , ..., 2.1489723 ,
2.334958 , 2.5260453 ],
[ -0.9514227 , -0.8912209 , -0.42839706, ..., -0.7747984 ,
-0.88131857, -1.1858144 ],
[ -1.6392798 , -1.8752912 , -1.3590232 , ..., -0.71193075,
-0.46426392, -0.79004574],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.117433 , -8.756327 , -8.3021965 , ..., 2.6770406 ,
2.9165506 , 2.619464 ],
[ -0.8334371 , -0.83099353, -0.7541424 , ..., -1.08567 ,
-0.3741293 , -0.14404964],
[ -1.1120342 , -1.2024816 , -1.1394764 , ..., -1.3766332 ,
-1.0821438 , -0.808341 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -8.535436 , -8.446434 , -8.433824 , ..., 1.7634034 ,
1.9151568 , 2.234317 ],
[ -0.73761785, -1.0498412 , -1.1566993 , ..., -1.9087753 ,
-1.4135947 , -0.43615532],
[ -3.1867118 , -1.3442845 , -0.6116022 , ..., -1.2935882 ,
-1.5357218 , -1.4339061 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
...,
[[ -8.752668 , -8.795702 , -9.234339 , ..., 1.4543443 ,
1.308625 , 1.8872037 ],
[ -0.8426125 , -0.5590738 , -0.64102423, ..., -0.2699995 ,
-0.34920692, -0.23712015],
[ -0.49331522, -1.201171 , -1.1223062 , ..., -1.4179659 ,
-0.8049512 , -0.8301816 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.640016 , -9.385439 , -9.126853 , ..., 2.6549459 ,
2.8241823 , 3.1816008 ],
[ -1.9176644 , -1.4502705 , -0.9663389 , ..., -0.6905961 ,
-0.6539016 , -0.5099206 ],
[ -1.4909953 , -1.6704174 , -1.1156216 , ..., -1.4263935 ,
-0.81988144, -1.1464152 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[-11.273719 , -10.959808 , -10.497703 , ..., 1.7455099 ,
1.9854834 , 2.464494 ],
[ -2.0011683 , -1.4769925 , -0.9517286 , ..., -0.25176716,
-0.5848155 , -0.5404253 ],
[ -1.4961203 , -1.4866172 , -1.359714 , ..., -1.1029644 ,
-1.4994144 , -0.89810133],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]]], dtype=float32)>
}, PerReplica:{
0: <tf.Tensor: shape=(8, 19), dtype=int32, numpy=
array([[ 0, 500, 7, 1030, 2607, 29, 108, 2559, 4196, 4131, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 70, 75, 2098, 14, 1025, 228, 26, 1178, 8, 5,
3233, 4131, 0, 0, 0, 0, 0, 0],
[ 0, 12, 1849, 4117, 3926, 156, 64, 10, 932, 29, 108,
302, 4131, 0, 0, 0, 0, 0, 0],
[ 0, 1, 3220, 4196, 4130, 2585, 1846, 1, 418, 45, 10,
164, 4131, 0, 0, 0, 0, 0, 0],
[ 0, 1, 426, 56, 215, 1333, 173, 304, 4131, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 108, 2186, 290, 600, 22, 931, 266, 631, 56, 42,
2335, 4199, 4131, 0, 0, 0, 0, 0],
[ 0, 23, 2684, 121, 5, 88, 4185, 2864, 99, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 4, 486, 471, 12, 5, 69, 30, 1286, 26, 495,
397, 332, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
1: <tf.Tensor: shape=(8, 19), dtype=int32, numpy=
array([[ 0, 20, 701, 16, 2598, 442, 20, 502, 2628, 494, 1363,
1566, 0, 0, 0, 0, 0, 0, 0],
[ 0, 399, 26, 356, 52, 2941, 4131, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 1, 575, 1465, 4117, 3062, 4130, 25, 22, 2190, 2940,
4193, 4131, 0, 0, 0, 0, 0, 0],
[ 0, 9, 97, 3737, 17, 59, 3619, 474, 1846, 4196, 4131,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 84, 1332, 177, 3837, 4117, 939, 1316, 4200, 4131, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 3, 123, 2750, 4117, 3, 1462, 4117, 27, 5, 875,
4131, 0, 0, 0, 0, 0, 0, 0],
[ 0, 3019, 1525, 4007, 592, 664, 16, 289, 11, 51, 4028,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 2321, 4117, 25, 59, 1002, 18, 365, 150, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
2: <tf.Tensor: shape=(8, 19), dtype=int32, numpy=
array([[ 0, 143, 12, 108, 539, 7, 34, 67, 25, 2746, 4148,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 9, 2373, 195, 496, 190, 4, 3921, 74, 2677, 1078,
45, 1, 532, 19, 23, 140, 4131, 0],
[ 0, 23, 18, 4026, 456, 2204, 15, 17, 35, 831, 470,
524, 0, 0, 0, 0, 0, 0, 0],
[ 0, 5, 512, 2, 36, 18, 724, 21, 20, 688, 4131,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 78, 68, 36, 89, 20, 157, 1279, 1855, 1910, 611,
528, 747, 8, 139, 2423, 4200, 4131, 0],
[ 0, 3, 28, 12, 3, 582, 4131, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 92, 2640, 186, 4, 1235, 10, 1718, 459, 4131, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 614, 1674, 2151, 492, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
3: <tf.Tensor: shape=(8, 19), dtype=int32, numpy=
array([[ 0, 1044, 4130, 25, 1, 2312, 269, 1781, 4117, 21, 191,
4079, 4131, 0, 0, 0, 0, 0, 0],
[ 0, 35, 2797, 4117, 1290, 4196, 4130, 1517, 737, 12, 1411,
2873, 4182, 4131, 0, 0, 0, 0, 0],
[ 0, 49, 641, 1525, 3145, 15, 25, 23, 411, 1628, 403,
4131, 0, 0, 0, 0, 0, 0, 0],
[ 0, 9, 2824, 335, 646, 25, 13, 3178, 633, 2679, 29,
5, 916, 34, 1558, 181, 4182, 4131, 0],
[ 0, 92, 112, 4025, 4182, 4118, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 37, 273, 280, 1296, 19, 23, 140, 4131, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 26, 3288, 15, 493, 1239, 1424, 117, 37, 5, 517,
266, 138, 4131, 0, 0, 0, 0, 0],
[ 0, 932, 29, 1, 1383, 416, 4117, 12, 4, 1620, 1170,
1737, 420, 1138, 4117, 2, 5, 1785, 4131]], dtype=int32)>
}, PerReplica:{
0: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([267, 225, 193, 132, 140, 159, 105, 137], dtype=int32)>,
1: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([166, 126, 146, 132, 135, 180, 184, 163], dtype=int32)>,
2: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([137, 239, 157, 172, 201, 112, 127, 124], dtype=int32)>,
3: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([200, 174, 151, 185, 165, 176, 204, 209], dtype=int32)>
}, PerReplica:{
0: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([ 9, 12, 12, 12, 8, 13, 8, 12], dtype=int32)>,
1: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([11, 6, 12, 10, 9, 11, 10, 8], dtype=int32)>,
2: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([10, 17, 11, 10, 17, 6, 9, 4], dtype=int32)>,
3: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([12, 13, 11, 17, 5, 8, 13, 18], dtype=int32)>
}, PerReplica:{
0: <tf.Tensor: shape=(8, 18), dtype=int32, numpy=
array([[ 500, 7, 1030, 2607, 29, 108, 2559, 4196, 4131, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 70, 75, 2098, 14, 1025, 228, 26, 1178, 8, 5, 3233,
4131, 0, 0, 0, 0, 0, 0],
[ 12, 1849, 4117, 3926, 156, 64, 10, 932, 29, 108, 302,
4131, 0, 0, 0, 0, 0, 0],
[ 1, 3220, 4196, 4130, 2585, 1846, 1, 418, 45, 10, 164,
4131, 0, 0, 0, 0, 0, 0],
[ 1, 426, 56, 215, 1333, 173, 304, 4131, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 108, 2186, 290, 600, 22, 931, 266, 631, 56, 42, 2335,
4199, 4131, 0, 0, 0, 0, 0],
[ 23, 2684, 121, 5, 88, 4185, 2864, 99, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 4, 486, 471, 12, 5, 69, 30, 1286, 26, 495, 397,
332, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
1: <tf.Tensor: shape=(8, 18), dtype=int32, numpy=
array([[ 20, 701, 16, 2598, 442, 20, 502, 2628, 494, 1363, 1566,
0, 0, 0, 0, 0, 0, 0],
[ 399, 26, 356, 52, 2941, 4131, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 1, 575, 1465, 4117, 3062, 4130, 25, 22, 2190, 2940, 4193,
4131, 0, 0, 0, 0, 0, 0],
[ 9, 97, 3737, 17, 59, 3619, 474, 1846, 4196, 4131, 0,
0, 0, 0, 0, 0, 0, 0],
[ 84, 1332, 177, 3837, 4117, 939, 1316, 4200, 4131, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 3, 123, 2750, 4117, 3, 1462, 4117, 27, 5, 875, 4131,
0, 0, 0, 0, 0, 0, 0],
[3019, 1525, 4007, 592, 664, 16, 289, 11, 51, 4028, 0,
0, 0, 0, 0, 0, 0, 0],
[2321, 4117, 25, 59, 1002, 18, 365, 150, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
2: <tf.Tensor: shape=(8, 18), dtype=int32, numpy=
array([[ 143, 12, 108, 539, 7, 34, 67, 25, 2746, 4148, 0,
0, 0, 0, 0, 0, 0, 0],
[ 9, 2373, 195, 496, 190, 4, 3921, 74, 2677, 1078, 45,
1, 532, 19, 23, 140, 4131, 0],
[ 23, 18, 4026, 456, 2204, 15, 17, 35, 831, 470, 524,
0, 0, 0, 0, 0, 0, 0],
[ 5, 512, 2, 36, 18, 724, 21, 20, 688, 4131, 0,
0, 0, 0, 0, 0, 0, 0],
[ 78, 68, 36, 89, 20, 157, 1279, 1855, 1910, 611, 528,
747, 8, 139, 2423, 4200, 4131, 0],
[ 3, 28, 12, 3, 582, 4131, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 92, 2640, 186, 4, 1235, 10, 1718, 459, 4131, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 614, 1674, 2151, 492, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
3: <tf.Tensor: shape=(8, 18), dtype=int32, numpy=
array([[1044, 4130, 25, 1, 2312, 269, 1781, 4117, 21, 191, 4079,
4131, 0, 0, 0, 0, 0, 0],
[ 35, 2797, 4117, 1290, 4196, 4130, 1517, 737, 12, 1411, 2873,
4182, 4131, 0, 0, 0, 0, 0],
[ 49, 641, 1525, 3145, 15, 25, 23, 411, 1628, 403, 4131,
0, 0, 0, 0, 0, 0, 0],
[ 9, 2824, 335, 646, 25, 13, 3178, 633, 2679, 29, 5,
916, 34, 1558, 181, 4182, 4131, 0],
[ 92, 112, 4025, 4182, 4118, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 37, 273, 280, 1296, 19, 23, 140, 4131, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 26, 3288, 15, 493, 1239, 1424, 117, 37, 5, 517, 266,
138, 4131, 0, 0, 0, 0, 0],
[ 932, 29, 1, 1383, 416, 4117, 12, 4, 1620, 1170, 1737,
420, 1138, 4117, 2, 5, 1785, 4131]], dtype=int32)>
}))
input_signature: (
[TensorSpec(shape=(None, None, 240), dtype=tf.float32, name=None), TensorSpec(shape=(None, None), dtype=tf.int32, name=None), TensorSpec(shape=(None,), dtype=tf.int32, name=None), TensorSpec(shape=(None,), dtype=tf.int32, name=None), TensorSpec(shape=(None, None), dtype=tf.int32, name=None)])
Any idea what could be the issue here?
I have the following setup:
$ pip freeze | grep tensor
tensorboard==2.2.1
tensorboard-plugin-wit==1.6.0.post3
tensorflow-datasets==3.1.0
tensorflow-estimator==2.2.0
tensorflow-gpu==2.2.0
tensorflow-metadata==0.22.0
warprnnt-tensorflow==0.1
I noticed that FLAGS.gpus is None which leads to MirroredStrategy(devices=gpu_names) where gpu_names is None. I am not sure if this has something to do with the issue.
Using only one GPU resolves this issue (see https://github.com/noahchalifour/rnnt-speech-recognition/issues/18#issuecomment-633862788). But I wouldn't call it "resolved" yet.
@BuaaAlban @prajwaljpj Were you able to run it on multiple GPUs yet?
No, and it does't converge
I'm able to run it on 8 v100 GPU in single instance, using TF2.2.0 according to https://github.com/tensorflow/tensorflow/issues/29911 .
The problem now is that it does't converge, the Loss is around 100 after 40 epochs, the batch size is 64, training data is common_voice_data/cv-corpus-6.1-2020-12-11_zh-CN.
