rnnt-speech-recognition
rnnt-speech-recognition copied to clipboard
Published
20 hours ago •
noahchalifour
noahchalifour
Multi-GPU training is not working
I have a machine with 2x Nvidia RTX 2080 Ti 8 Core Intel i7 processor 32Gb of RAM
The training code (non-Docker version) when CUDA_VISIBLE_DEVICES=0,1 causes a memory leak in eval_step.
python run_common_voice.py --mode train --data_dir
Performing evaluation. [949/1811]
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:38.737240 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:38.740701 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:38.743986 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:38.747186 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
2020-04-21 00:39:43.431398: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-04-21 00:39:44.193788: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
WARNING:tensorflow:Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_v2 inside a tf.function to get the best perf$
rmance.
W0421 00:39:49.856330 140487075895104 mirrored_strategy.py:692] Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_$ 2 inside a tf.function to get the best performance.
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:49.859219 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:49.859964 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
WARNING:tensorflow:Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_v2 inside a tf.function to get the best perf$
rmance.
W0421 00:39:49.861165 140487075895104 mirrored_strategy.py:692] Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_$ 2 inside a tf.function to get the best performance.
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:49.863494 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:49.864265 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
WARNING:tensorflow:Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_v2 inside a tf.function to get the best perf$
rmance.
W0421 00:39:49.865403 140487075895104 mirrored_strategy.py:692] Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_$ 2 inside a tf.function to get the best performance.
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:49.867894 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0421 00:39:49.868691 140487075895104 cross_device_ops.py:439] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
WARNING:tensorflow:Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_v2 inside a tf.function to get the best perfo
rmance.
W0421 00:39:49.869868 140487075895104 mirrored_strategy.py:692] Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_v 2 inside a tf.function to get the best performance.
WARNING:tensorflow:Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_v2 inside a tf.function to get the best perfo
rmance.
W0421 00:40:01.864544 140487075895104 mirrored_strategy.py:692] Using MirroredStrategy eagerly has significant overhead currently. We will be working on improving this in the future, but for now please wrap call_for_each_replica or experimental_run or experimental_run_v 2 inside a tf.function to get the best performance.
WARNING:tensorflow:5 out of the last 5 calls to <function run_evaluate.
Is this a tensorflow issue?
I have same issue. My system is
RAM : 128GB GPU : GTX 1080ti * 4 OS : ubuntu 18.04 NVIDIA Driver : 440.82 CUDA : 10.1 CUDNN : 7.6.5 python : 3.6.9 tensorflow & tensorflow-gpu : 2.1.0 (And I do not change any param in run_common_voice.py)
When I run the run_common_voice.py code. These are shown.
At the 0th epoch Eval_step is running with retracing warning and then, I got the OOM error.
Disable evaluation at the 0th epoch. 2-1. When there is retracing warning (slow) Epoch: 0, Batch: 60, Global Step: 60, Step Time: 26.0310, Loss: 165.6244 2-2. When there is no retracing warning (fast) Epoch: 0, Batch: 62, Global Step: 62, Step Time: 6.3741, Loss: 164.6387
Then I get the OOM error after this line Epoch: 0, Batch: 226, Global Step: 226, Step Time: 5.9092, Loss: 142.7257 ...
I think some of the tf.function? affect to speed of the training.
Does the retracing warning have a connection with OOM error? --> If so, how can I solve the retracing warning? --> If not, how can I solve the OOM error?
Thank you
@nambee Did single GPU training work for you?
No it does not work.
To see the progress, I print some logs in 'run_evaluate' func which is inside of 'run_training' func. (I attach this code at the end of this comment. (I only added 'print' functions.)) After 432 batches, OOM error has occurred. (+ The total eval_dataset loop count is 486.)
CUDA_VISIBLE_DEVICE=1 python run_common_voice.py --mode train --data_dir english_data/feature
... tensorflow.org/api_docs/python/tf/function for more details. Performing evaluation.2-2 Performing evaluation.2-3 -------------------- [432] ------------------ Performing evaluation.2-1 eval_step : <tensorflow.python.eager.def_function.Function object at 0x7f6dac6b07b8> Type : eval_step : <class 'tensorflow.python.eager.def_function.Function'> input type : <class 'tuple'> Performing evaluation.2-1-1 Performing evaluation.2-1-2 Performing evaluation.2-1-3: Tensor("Identity:0", shape=(), dtype=float32, device=/job:localhost/replica:0/task:0/device:CPU:0) Performing evaluation.2-1-4: {'WER': <tf.Tensor 'Identity_1:0' shape=
dtype=float32>, 'Accuracy': <tf.Tensor 'Identity_2:0' shape= dtype=float32>, 'CER': <tf.Tensor 'Identity_3:0' shape= dtype=float32>} 2020-04-22 15:55:35.613508: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory ... (0) Resource exhausted: OOM when allocating tensor with shape[8,4088,303,37] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node transducer/dense_1/BiasAdd-0-TransposeNHWCToNCHW-LayoutOptimizer}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. [[replica_3/StringsByteSplit_1/RaggedGetItem/strided_slice_4/stack_1/_1212]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[8,4088,303,37] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node transducer/dense_1/BiasAdd-0-TransposeNHWCToNCHW-LayoutOptimizer}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
def run_evaluate(model,
optimizer,
loss_fn,
eval_dataset,
batch_size,
strategy,
metrics=[],
fp16_run=False,
gpus=[]):
@tf.function(experimental_relax_shapes=True)
def eval_step(dist_inputs):
def step_fn(inputs):
(mel_specs, pred_inp,
spec_lengths, label_lengths, labels) = inputs
outputs = model([mel_specs, pred_inp],
training=False)
loss = loss_fn(labels, outputs,
spec_lengths=spec_lengths,
label_lengths=label_lengths)
loss *= (1. / batch_size)
if fp16_run:
loss = optimizer.get_scaled_loss(loss)
if metrics is not None:
metric_results = run_metrics(mel_specs, labels,
metrics=metrics)
metric_results = {name: result * (1. / max(len(gpus), 1)) for name, result in metric_results.items()}
return loss, metric_results
print('Performing evaluation.2-1-1')
losses, metrics_results = strategy.experimental_run_v2(step_fn, args=(dist_inputs,))
print('Performing evaluation.2-1-2')
mean_loss = strategy.reduce(
tf.distribute.ReduceOp.SUM, losses, axis=0)
print('Performing evaluation.2-1-3:',mean_loss)
mean_metrics = {name: strategy.reduce(
tf.distribute.ReduceOp.SUM, result, axis=0) for name, result in metrics_results.items()}
print('Performing evaluation.2-1-4:',mean_metrics)
return mean_loss, mean_metrics
print('Performing evaluation.')
loss_object = tf.keras.metrics.Mean()
metric_objects = {fn.__name__: tf.keras.metrics.Mean() for fn in metrics}
print('Performing evaluation.2 ')
cnt = 0
for batch, inputs in enumerate(eval_dataset):
cnt = cnt +1
print('-------------------- ['+str(cnt)+'] ------------------')
print('Performing evaluation.2-1')
print('eval_step : ',eval_step)
print('Type : eval_step : ',type(eval_step))
print('input type : ',type(inputs))
loss, metrics_results = eval_step(inputs)
print('Performing evaluation.2-2')
loss_object(loss)
print('Performing evaluation.2-3')
for metric_name, metric_result in metrics_results.items():
metric_objects[metric_name](metric_result)
print('Performing evaluation.3')
metrics_final_results = {name: metric_object.result() for name, metric_object in metric_objects.items()}
print('Performing evaluation. finish')
return loss_object.result(), metrics_final_results
@nambee From this log, you can see that you are running out of gpu memory, reduce the batch size to 8 or lower should fix the problem. 2020-04-22 15:55:35.613508: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
But still it looks like due to eager execution, the memory requirement keeps growing and only at eval step. My system fails to allocate GPU memory after 19000 Batches at Epoc 0. @noahchalifour is there a way to fix this?
Oh, it's a different issue. sorry. I thought you end up with OOM error too.
Can you run the run_common_voice.py without the OOM error? I got the OOM error for eval_step and train_step too. (I disabled the eval step to see the train_step can work.)
Oh, it's a different issue. sorry. I thought you end up with OOM error too.
Can you run the run_common_voice.py without the OOM error? I got the OOM error for eval_step and train_step too. (I disabled the eval step to see the train_step can work.)
Yes it worked for me. Even though you use CUDA_VISIBLE_DEVICES=0 to specify one GPU you have to change the strategy = None in run_common_voice.py.
@prajwaljpj Thank you for your advice. Retracing errors are gone when I disable strategy. I still got the OOM error, I should reduce some factors. Again, Thank you!
@nambee Strategy part is not implemented for eval. If you see the training function there is a condition which implements strategy and experimental_run. You have to make a similar change for eval. also Try reducing batch size to 2.
@prajwaljpj Yes, I did that already. But I apply it only for small datasets. (Because I need feasibility now) I will expand it in the future. Thank you for your kind consideration.
Can someone please let me know if this is resolved in the latest commit? I do not have a multi GPU machine to test on. Thanks
Could this be related to https://github.com/noahchalifour/rnnt-speech-recognition/issues/29 ?
It does seem so.
First off, there seems to be an error, gpus is not defined at this point and run_evaluate() does not expose an argument gpus.
https://github.com/noahchalifour/rnnt-speech-recognition/blob/a0d972f5e407e465ad784c682fa4e72e33d8eefe/run_rnnt.py#L570
If I run the training with CUDA_VISIBLE_DEVICES=0 it does seem to work. However, running with multiple GPUs gives me the exception as described in https://github.com/noahchalifour/rnnt-speech-recognition/issues/29.
for completeness, click to expand full error log
/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/librosa/util/decorators.py:9: NumbaDeprecationWarning: An import was requested from a module that has moved location.
Import requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0.
from numba.decorators import jit as optional_jit
/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/librosa/util/decorators.py:9: NumbaDeprecationWarning: An import was requested from a module that has moved location.
Import of 'jit' requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0.
from numba.decorators import jit as optional_jit
2020-05-26 09:14:30.736191: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-26 09:14:30.748386: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.749173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:30.749232: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.750058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 1 with properties:
pciBusID: 0000:02:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:30.750112: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.750888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 2 with properties:
pciBusID: 0000:03:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:30.750927: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.751427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 3 with properties:
pciBusID: 0000:05:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:30.751570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-26 09:14:30.752638: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-26 09:14:30.753673: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-26 09:14:30.753866: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-26 09:14:30.754997: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-26 09:14:30.755618: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-26 09:14:30.757804: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-26 09:14:30.757899: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.759345: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.760097: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.760844: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.761589: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.762328: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.763068: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.763805: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:30.764521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0, 1, 2, 3
2020-05-26 09:14:30.764770: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-05-26 09:14:30.770070: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 4200000000 Hz
2020-05-26 09:14:30.770492: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560e6150fef0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-26 09:14:30.770507: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-05-26 09:14:31.020514: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.037961: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.041811: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.049635: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.050189: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560e60e72d20 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-26 09:14:31.050199: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-05-26 09:14:31.050203: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (1): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-05-26 09:14:31.050206: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (2): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-05-26 09:14:31.050209: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (3): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-05-26 09:14:31.051527: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.051949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:31.051989: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.052409: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 1 with properties:
pciBusID: 0000:02:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:31.052448: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.052867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 2 with properties:
pciBusID: 0000:03:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:31.052904: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.053326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 3 with properties:
pciBusID: 0000:05:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-26 09:14:31.053353: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-26 09:14:31.053366: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-26 09:14:31.053377: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-26 09:14:31.053387: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-26 09:14:31.053397: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-26 09:14:31.053407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-26 09:14:31.053418: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-26 09:14:31.053452: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.053895: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.054339: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.054782: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.055227: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.055669: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.056126: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.056579: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.057003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0, 1, 2, 3
2020-05-26 09:14:31.057025: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-26 09:14:31.059325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-26 09:14:31.059335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 1 2 3
2020-05-26 09:14:31.059340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N Y Y Y
2020-05-26 09:14:31.059344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 1: Y N Y Y
2020-05-26 09:14:31.059347: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 2: Y Y N Y
2020-05-26 09:14:31.059350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 3: Y Y Y N
2020-05-26 09:14:31.060102: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.060567: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.061033: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.061486: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.061942: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.062368: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9449 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-05-26 09:14:31.062688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.063131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10161 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2020-05-26 09:14:31.063473: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.064690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10161 MB memory) -> physical GPU (device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1)
2020-05-26 09:14:31.065011: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 09:14:31.065455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10161 MB memory) -> physical GPU (device: 3, name: GeForce GTX 1080 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
4 Physical GPU, 4 Logical GPUs
WARNING:tensorflow:From /home/sfalk/tmp/rnnt-speech-recognition/model.py:59: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
W0526 09:14:32.108052 140106746382080 deprecation.py:317] From /home/sfalk/tmp/rnnt-speech-recognition/model.py:59: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c9e97d820>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.108385 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c9e97d820>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:From /home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/ops/rnn_cell_impl.py:962: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.
W0526 09:14:32.109819 140106746382080 deprecation.py:317] From /home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/ops/rnn_cell_impl.py:962: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c9009e730>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.227335 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c9009e730>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c4811f9d0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.490125 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c4811f9d0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c48086070>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.669947 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c48086070>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c383bc4f0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.804272 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c383bc4f0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c383a7d00>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:32.951039 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c383a7d00>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c3830e190>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:33.074690 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c3830e190>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c382f8250>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:33.202479 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c382f8250>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c381bd820>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:33.890956 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c381bd820>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
WARNING:tensorflow:<tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c086b71f0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
W0526 09:14:34.015121 140106746382080 rnn_cell_impl.py:909] <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x7f6c086b71f0>: Note that this cell is not optimized for performance. Please use tf.contrib.cudnn_rnn.CudnnLSTM for better performance on GPU.
I0526 09:14:34.344151 140106746382080 run_rnnt.py:490] Using word-piece encoder with vocab size: 4341
Model: "encoder"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, None, 240)] 0
_________________________________________________________________
batch_normalization (BatchNo (None, None, 240) 960
_________________________________________________________________
rnn (RNN) (None, None, 640) 8527872
_________________________________________________________________
dropout (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization (LayerNo (None, None, 640) 1280
_________________________________________________________________
rnn_1 (RNN) (None, None, 640) 11804672
_________________________________________________________________
dropout_1 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_1 (Layer (None, None, 640) 1280
_________________________________________________________________
time_reduction (TimeReductio (None, None, 1280) 0
_________________________________________________________________
rnn_2 (RNN) (None, None, 640) 17047552
_________________________________________________________________
dropout_2 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_2 (Layer (None, None, 640) 1280
_________________________________________________________________
rnn_3 (RNN) (None, None, 640) 11804672
_________________________________________________________________
dropout_3 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_3 (Layer (None, None, 640) 1280
_________________________________________________________________
rnn_4 (RNN) (None, None, 640) 11804672
_________________________________________________________________
dropout_4 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_4 (Layer (None, None, 640) 1280
_________________________________________________________________
rnn_5 (RNN) (None, None, 640) 11804672
_________________________________________________________________
dropout_5 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_5 (Layer (None, None, 640) 1280
_________________________________________________________________
rnn_6 (RNN) (None, None, 640) 11804672
_________________________________________________________________
dropout_6 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_6 (Layer (None, None, 640) 1280
_________________________________________________________________
rnn_7 (RNN) (None, None, 640) 11804672
_________________________________________________________________
dropout_7 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_7 (Layer (None, None, 640) 1280
=================================================================
Total params: 96,414,656
Trainable params: 96,414,176
Non-trainable params: 480
_________________________________________________________________
Model: "prediction_network"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_2 (InputLayer) [(None, None)] 0
_________________________________________________________________
embedding (Embedding) (None, None, 500) 2170500
_________________________________________________________________
rnn_8 (RNN) (None, None, 640) 10657792
_________________________________________________________________
dropout_8 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_8 (Layer (None, None, 640) 1280
_________________________________________________________________
rnn_9 (RNN) (None, None, 640) 11804672
_________________________________________________________________
dropout_9 (Dropout) (None, None, 640) 0
_________________________________________________________________
layer_normalization_9 (Layer (None, None, 640) 1280
=================================================================
Total params: 24,635,524
Trainable params: 24,635,524
Non-trainable params: 0
_________________________________________________________________
Model: "transducer"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
mel_specs (InputLayer) [(None, None, 240)] 0
__________________________________________________________________________________________________
pred_inp (InputLayer) [(None, None)] 0
__________________________________________________________________________________________________
encoder (Model) (None, None, 640) 96414656 mel_specs[0][0]
__________________________________________________________________________________________________
prediction_network (Model) (None, None, 640) 24635524 pred_inp[0][0]
__________________________________________________________________________________________________
tf_op_layer_ExpandDims (TensorF [(None, None, 1, 640 0 encoder[1][0]
__________________________________________________________________________________________________
tf_op_layer_ExpandDims_1 (Tenso [(None, 1, None, 640 0 prediction_network[1][0]
__________________________________________________________________________________________________
tf_op_layer_AddV2 (TensorFlowOp [(None, None, None, 0 tf_op_layer_ExpandDims[0][0]
tf_op_layer_ExpandDims_1[0][0]
__________________________________________________________________________________________________
dense (Dense) (None, None, None, 6 410240 tf_op_layer_AddV2[0][0]
__________________________________________________________________________________________________
dense_1 (Dense) (None, None, None, 4 2782581 dense[0][0]
==================================================================================================
Total params: 124,243,001
Trainable params: 124,242,521
Non-trainable params: 480
__________________________________________________________________________________________________
Starting training.
Performing evaluation.
Traceback (most recent call last):
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2292, in _convert_inputs_to_signature
flatten_inputs[index] = ops.convert_to_tensor(
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1341, in convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 321, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 261, in constant
return _constant_impl(value, dtype, shape, name, verify_shape=False,
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 270, in _constant_impl
t = convert_to_eager_tensor(value, ctx, dtype)
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 96, in convert_to_eager_tensor
return ops.EagerTensor(value, ctx.device_name, dtype)
ValueError: Attempt to convert a value (PerReplica:{
0: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-9.8887777e+00, -9.5391264e+00, -9.2146311e+00, ...,
1.4807711e+00, 1.4137149e+00, 1.5833356e+00],
[-2.5297828e+00, -1.0314496e+00, -4.4551528e-01, ...,
-8.5550594e-01, -3.8671780e-01, -6.2595654e-01],
[-9.2890608e-01, -9.3925929e-01, -1.0737282e+00, ...,
-4.6040058e-01, -1.3226795e-01, -4.6705770e-01],
...,
[-8.9524627e-02, -1.3095784e-01, 4.4763446e-02, ...,
-7.6179504e-03, 3.1356859e-01, 1.3805485e-01],
[-1.0855615e-01, -3.6668968e-01, -3.5269606e-01, ...,
-1.6952515e-01, -4.3339968e-01, -2.3297167e-01],
[-3.4607446e-01, -4.6576285e-01, -2.4114418e-01, ...,
-3.9931583e-01, -6.5470409e-01, -5.9117317e-02]],
[[-1.0188073e+01, -9.7674351e+00, -9.2495003e+00, ...,
2.7845190e+00, 3.0497322e+00, 2.9723659e+00],
[-5.8545446e-01, -9.0612173e-01, -1.4650891e+00, ...,
-1.2318707e+00, -1.2984281e+00, -1.2217040e+00],
[-4.9245834e-01, -6.1498523e-01, -9.3534470e-01, ...,
-1.1765385e+00, -1.4464822e+00, -6.3945484e-01],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
[[-9.7931051e+00, -9.2576466e+00, -8.7187033e+00, ...,
2.6441813e+00, 2.5935564e+00, 2.7019000e+00],
[-1.1656429e+00, -2.4396741e-01, 2.2748601e-01, ...,
-1.1423826e+00, -1.9891844e+00, -1.8549285e+00],
[-1.1924456e+00, -1.5766211e+00, -1.9995271e+00, ...,
-6.9014835e-01, -1.1843119e+00, -1.7883348e+00],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
...,
[[-9.2142944e+00, -8.7442112e+00, -8.0906887e+00, ...,
2.5637565e+00, 2.6928663e+00, 2.9714472e+00],
[-1.4353551e+00, -1.3912488e+00, -1.1410775e+00, ...,
-6.8289948e-01, -7.4661875e-01, -9.4346857e-01],
[-1.3514730e+00, -1.3447710e+00, -1.1697345e+00, ...,
-8.4204197e-01, -6.4308357e-01, -6.6116714e-01],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
[[-9.4380770e+00, -9.2610064e+00, -9.3578424e+00, ...,
1.3151474e+00, 1.3194189e+00, 1.6185846e+00],
[-1.4748455e+00, -9.6358955e-01, -7.5807011e-01, ...,
-1.2112403e+00, -5.9023190e-01, -2.0361996e-01],
[-1.2093054e+00, -1.8930241e+00, -1.3843757e+00, ...,
-3.7091255e-01, -6.0484600e-01, -1.9562192e+00],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
[[-9.2918978e+00, -9.2315750e+00, -8.9223146e+00, ...,
2.7278879e+00, 2.8609688e+00, 2.7423246e+00],
[-5.9187198e-01, -1.8667960e-01, 2.1728778e-01, ...,
-9.7960901e-01, -2.1147966e-01, -8.8720083e-01],
[-8.6280262e-01, -8.0901504e-01, -5.0346565e-01, ...,
-7.1112490e-01, -1.2980080e-01, 3.8867188e-01],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]]], dtype=float32)>,
1: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-10.0592985 , -9.730883 , -9.271258 , ..., 2.8219585 ,
2.6701574 , 3.145619 ],
[ -0.9512273 , -0.57793355, -0.37342393, ..., -0.36957645,
-0.58688927, -0.6034956 ],
[ -1.0401095 , -1.0510697 , -1.1356157 , ..., -0.2553835 ,
-0.40022898, -1.2278776 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.3389225 , -9.18263 , -8.9984045 , ..., 2.095945 ,
2.1119766 , 2.3430693 ],
[ -0.9921645 , -0.9324732 , -0.67674816, ..., -0.8709469 ,
-0.87235546, -0.8106818 ],
[ -2.0774326 , -2.127688 , -1.135358 , ..., -0.97692585,
-1.1275706 , -0.88771343],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[-10.804321 , -10.686319 , -10.408899 , ..., 2.3665676 ,
2.5379932 , 2.420307 ],
[ -1.1987064 , -1.067873 , -0.9529356 , ..., -0.30030394,
-0.12877178, -0.16108751],
[ -0.45661426, -0.7980306 , -1.1557912 , ..., -1.5581841 ,
-0.4684105 , -0.44000912],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
...,
[[ -9.445232 , -9.460578 , -9.285489 , ..., 1.8623145 ,
1.9304647 , 2.195097 ],
[ -1.0719146 , -1.0908513 , -0.9582559 , ..., -0.99088955,
-0.6627326 , -0.6685376 ],
[ -1.8296288 , -1.8624439 , -1.321793 , ..., -0.70586205,
-0.97963333, -1.346128 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.328621 , -9.586381 , -9.817234 , ..., 3.6245484 ,
3.8879015 , 3.8824553 ],
[ -2.2337832 , -1.1850247 , -0.43120062, ..., -1.3237286 ,
-0.8165655 , -0.5502229 ],
[ -4.9042144 , -1.1433594 , -0.3725183 , ..., 0.19899416,
0.02039671, 0.09885693],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.299686 , -8.799611 , -8.434992 , ..., 3.097272 ,
3.1601548 , 3.1777468 ],
[ -1.1876402 , -1.0250216 , -0.64193165, ..., -0.32103252,
-0.06696558, -0.6231766 ],
[ -0.5330059 , -0.6796452 , -0.77790976, ..., -0.9301033 ,
-0.8343253 , -0.95349455],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]]], dtype=float32)>,
2: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-10.20004 , -9.797416 , -9.144037 , ..., 2.1522975 ,
2.177644 , 2.4891276 ],
[ -0.6829766 , -1.0327927 , -1.9740211 , ..., -1.109499 ,
-1.2662077 , -1.4992175 ],
[ -0.50412035, -0.75838935, -1.4640208 , ..., -1.1935134 ,
-1.90482 , -1.4951177 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.057947 , -8.835877 , -8.432611 , ..., 2.7456422 ,
2.6064804 , 2.9408405 ],
[ -0.7960679 , -0.7536596 , -0.56650114, ..., -0.9725952 ,
-0.9330125 , -0.8314767 ],
[ -1.273035 , -1.0706124 , -0.8415742 , ..., -1.0601668 ,
-1.3329258 , -2.0243492 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -8.598597 , -8.078391 , -7.316219 , ..., 3.475574 ,
3.5632377 , 3.4421172 ],
[ -0.15228844, 0.27448 , 0.84499 , ..., -1.2552052 ,
-0.435256 , -0.70891285],
[ -0.08234251, -0.10229468, -0.20772624, ..., 0.49269104,
-0.33324862, -0.30197382],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
...,
[[-10.674739 , -10.312547 , -9.784449 , ..., 1.9142385 ,
2.0289752 , 2.207048 ],
[ -2.3002534 , -1.4031292 , -0.7912599 , ..., 0.46009445,
0.26866436, 0.23870277],
[ -0.8622447 , -0.8987092 , -0.7318269 , ..., -0.51101494,
0.02473354, 0.04263115],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[-11.913138 , -11.890651 , -11.718769 , ..., -0.6516266 ,
-0.11420345, -0.2889433 ],
[ -2.3279111 , -0.83327794, -0.38966238, ..., -0.18499136,
-0.31412745, -0.78526115],
[ -1.2105832 , -0.65170074, -0.30061066, ..., -0.5946727 ,
-0.5343251 , -0.26280165],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.108984 , -8.4880495 , -7.7428994 , ..., 3.0713623 ,
3.0152838 , 3.4003892 ],
[ -0.19017792, -0.29030776, -0.33207464, ..., -0.0172677 ,
-0.24603939, -0.9277673 ],
[ -0.6175966 , -0.8139999 , -1.1248744 , ..., -0.649323 ,
-0.6167655 , -0.5438595 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]]], dtype=float32)>,
3: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[ -8.988267 , -8.965929 , -8.728106 , ..., 2.1489723 ,
2.334958 , 2.5260453 ],
[ -0.9514227 , -0.8912209 , -0.42839706, ..., -0.7747984 ,
-0.88131857, -1.1858144 ],
[ -1.6392798 , -1.8752912 , -1.3590232 , ..., -0.71193075,
-0.46426392, -0.79004574],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.117433 , -8.756327 , -8.3021965 , ..., 2.6770406 ,
2.9165506 , 2.619464 ],
[ -0.8334371 , -0.83099353, -0.7541424 , ..., -1.08567 ,
-0.3741293 , -0.14404964],
[ -1.1120342 , -1.2024816 , -1.1394764 , ..., -1.3766332 ,
-1.0821438 , -0.808341 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -8.535436 , -8.446434 , -8.433824 , ..., 1.7634034 ,
1.9151568 , 2.234317 ],
[ -0.73761785, -1.0498412 , -1.1566993 , ..., -1.9087753 ,
-1.4135947 , -0.43615532],
[ -3.1867118 , -1.3442845 , -0.6116022 , ..., -1.2935882 ,
-1.5357218 , -1.4339061 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
...,
[[ -8.752668 , -8.795702 , -9.234339 , ..., 1.4543443 ,
1.308625 , 1.8872037 ],
[ -0.8426125 , -0.5590738 , -0.64102423, ..., -0.2699995 ,
-0.34920692, -0.23712015],
[ -0.49331522, -1.201171 , -1.1223062 , ..., -1.4179659 ,
-0.8049512 , -0.8301816 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.640016 , -9.385439 , -9.126853 , ..., 2.6549459 ,
2.8241823 , 3.1816008 ],
[ -1.9176644 , -1.4502705 , -0.9663389 , ..., -0.6905961 ,
-0.6539016 , -0.5099206 ],
[ -1.4909953 , -1.6704174 , -1.1156216 , ..., -1.4263935 ,
-0.81988144, -1.1464152 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[-11.273719 , -10.959808 , -10.497703 , ..., 1.7455099 ,
1.9854834 , 2.464494 ],
[ -2.0011683 , -1.4769925 , -0.9517286 , ..., -0.25176716,
-0.5848155 , -0.5404253 ],
[ -1.4961203 , -1.4866172 , -1.359714 , ..., -1.1029644 ,
-1.4994144 , -0.89810133],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]]], dtype=float32)>
}) with an unsupported type (<class 'tensorflow.python.distribute.values.PerReplica'>) to a Tensor.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "run_rnnt.py", line 586, in <module>
app.run(main)
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "run_rnnt.py", line 532, in main
run_training(
File "run_rnnt.py", line 347, in run_training
checkpoint_model()
File "run_rnnt.py", line 304, in checkpoint_model
eval_loss, eval_metrics_results = run_evaluate(
File "run_rnnt.py", line 433, in run_evaluate
loss, metrics_results = eval_step(inputs)
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
result = self._call(*args, **kwds)
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 647, in _call
self._stateful_fn._function_spec.canonicalize_function_inputs( # pylint: disable=protected-access
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2235, in canonicalize_function_inputs
inputs = _convert_inputs_to_signature(
File "/home/sfalk/miniconda3/envs/rnnt/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2296, in _convert_inputs_to_signature
raise ValueError("When input_signature is provided, all inputs to "
ValueError: When input_signature is provided, all inputs to the Python function must be convertible to tensors:
inputs: (
(PerReplica:{
0: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-9.8887777e+00, -9.5391264e+00, -9.2146311e+00, ...,
1.4807711e+00, 1.4137149e+00, 1.5833356e+00],
[-2.5297828e+00, -1.0314496e+00, -4.4551528e-01, ...,
-8.5550594e-01, -3.8671780e-01, -6.2595654e-01],
[-9.2890608e-01, -9.3925929e-01, -1.0737282e+00, ...,
-4.6040058e-01, -1.3226795e-01, -4.6705770e-01],
...,
[-8.9524627e-02, -1.3095784e-01, 4.4763446e-02, ...,
-7.6179504e-03, 3.1356859e-01, 1.3805485e-01],
[-1.0855615e-01, -3.6668968e-01, -3.5269606e-01, ...,
-1.6952515e-01, -4.3339968e-01, -2.3297167e-01],
[-3.4607446e-01, -4.6576285e-01, -2.4114418e-01, ...,
-3.9931583e-01, -6.5470409e-01, -5.9117317e-02]],
[[-1.0188073e+01, -9.7674351e+00, -9.2495003e+00, ...,
2.7845190e+00, 3.0497322e+00, 2.9723659e+00],
[-5.8545446e-01, -9.0612173e-01, -1.4650891e+00, ...,
-1.2318707e+00, -1.2984281e+00, -1.2217040e+00],
[-4.9245834e-01, -6.1498523e-01, -9.3534470e-01, ...,
-1.1765385e+00, -1.4464822e+00, -6.3945484e-01],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
[[-9.7931051e+00, -9.2576466e+00, -8.7187033e+00, ...,
2.6441813e+00, 2.5935564e+00, 2.7019000e+00],
[-1.1656429e+00, -2.4396741e-01, 2.2748601e-01, ...,
-1.1423826e+00, -1.9891844e+00, -1.8549285e+00],
[-1.1924456e+00, -1.5766211e+00, -1.9995271e+00, ...,
-6.9014835e-01, -1.1843119e+00, -1.7883348e+00],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
...,
[[-9.2142944e+00, -8.7442112e+00, -8.0906887e+00, ...,
2.5637565e+00, 2.6928663e+00, 2.9714472e+00],
[-1.4353551e+00, -1.3912488e+00, -1.1410775e+00, ...,
-6.8289948e-01, -7.4661875e-01, -9.4346857e-01],
[-1.3514730e+00, -1.3447710e+00, -1.1697345e+00, ...,
-8.4204197e-01, -6.4308357e-01, -6.6116714e-01],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
[[-9.4380770e+00, -9.2610064e+00, -9.3578424e+00, ...,
1.3151474e+00, 1.3194189e+00, 1.6185846e+00],
[-1.4748455e+00, -9.6358955e-01, -7.5807011e-01, ...,
-1.2112403e+00, -5.9023190e-01, -2.0361996e-01],
[-1.2093054e+00, -1.8930241e+00, -1.3843757e+00, ...,
-3.7091255e-01, -6.0484600e-01, -1.9562192e+00],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],
[[-9.2918978e+00, -9.2315750e+00, -8.9223146e+00, ...,
2.7278879e+00, 2.8609688e+00, 2.7423246e+00],
[-5.9187198e-01, -1.8667960e-01, 2.1728778e-01, ...,
-9.7960901e-01, -2.1147966e-01, -8.8720083e-01],
[-8.6280262e-01, -8.0901504e-01, -5.0346565e-01, ...,
-7.1112490e-01, -1.2980080e-01, 3.8867188e-01],
...,
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]]], dtype=float32)>,
1: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-10.0592985 , -9.730883 , -9.271258 , ..., 2.8219585 ,
2.6701574 , 3.145619 ],
[ -0.9512273 , -0.57793355, -0.37342393, ..., -0.36957645,
-0.58688927, -0.6034956 ],
[ -1.0401095 , -1.0510697 , -1.1356157 , ..., -0.2553835 ,
-0.40022898, -1.2278776 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.3389225 , -9.18263 , -8.9984045 , ..., 2.095945 ,
2.1119766 , 2.3430693 ],
[ -0.9921645 , -0.9324732 , -0.67674816, ..., -0.8709469 ,
-0.87235546, -0.8106818 ],
[ -2.0774326 , -2.127688 , -1.135358 , ..., -0.97692585,
-1.1275706 , -0.88771343],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[-10.804321 , -10.686319 , -10.408899 , ..., 2.3665676 ,
2.5379932 , 2.420307 ],
[ -1.1987064 , -1.067873 , -0.9529356 , ..., -0.30030394,
-0.12877178, -0.16108751],
[ -0.45661426, -0.7980306 , -1.1557912 , ..., -1.5581841 ,
-0.4684105 , -0.44000912],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
...,
[[ -9.445232 , -9.460578 , -9.285489 , ..., 1.8623145 ,
1.9304647 , 2.195097 ],
[ -1.0719146 , -1.0908513 , -0.9582559 , ..., -0.99088955,
-0.6627326 , -0.6685376 ],
[ -1.8296288 , -1.8624439 , -1.321793 , ..., -0.70586205,
-0.97963333, -1.346128 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.328621 , -9.586381 , -9.817234 , ..., 3.6245484 ,
3.8879015 , 3.8824553 ],
[ -2.2337832 , -1.1850247 , -0.43120062, ..., -1.3237286 ,
-0.8165655 , -0.5502229 ],
[ -4.9042144 , -1.1433594 , -0.3725183 , ..., 0.19899416,
0.02039671, 0.09885693],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.299686 , -8.799611 , -8.434992 , ..., 3.097272 ,
3.1601548 , 3.1777468 ],
[ -1.1876402 , -1.0250216 , -0.64193165, ..., -0.32103252,
-0.06696558, -0.6231766 ],
[ -0.5330059 , -0.6796452 , -0.77790976, ..., -0.9301033 ,
-0.8343253 , -0.95349455],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]]], dtype=float32)>,
2: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[-10.20004 , -9.797416 , -9.144037 , ..., 2.1522975 ,
2.177644 , 2.4891276 ],
[ -0.6829766 , -1.0327927 , -1.9740211 , ..., -1.109499 ,
-1.2662077 , -1.4992175 ],
[ -0.50412035, -0.75838935, -1.4640208 , ..., -1.1935134 ,
-1.90482 , -1.4951177 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.057947 , -8.835877 , -8.432611 , ..., 2.7456422 ,
2.6064804 , 2.9408405 ],
[ -0.7960679 , -0.7536596 , -0.56650114, ..., -0.9725952 ,
-0.9330125 , -0.8314767 ],
[ -1.273035 , -1.0706124 , -0.8415742 , ..., -1.0601668 ,
-1.3329258 , -2.0243492 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -8.598597 , -8.078391 , -7.316219 , ..., 3.475574 ,
3.5632377 , 3.4421172 ],
[ -0.15228844, 0.27448 , 0.84499 , ..., -1.2552052 ,
-0.435256 , -0.70891285],
[ -0.08234251, -0.10229468, -0.20772624, ..., 0.49269104,
-0.33324862, -0.30197382],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
...,
[[-10.674739 , -10.312547 , -9.784449 , ..., 1.9142385 ,
2.0289752 , 2.207048 ],
[ -2.3002534 , -1.4031292 , -0.7912599 , ..., 0.46009445,
0.26866436, 0.23870277],
[ -0.8622447 , -0.8987092 , -0.7318269 , ..., -0.51101494,
0.02473354, 0.04263115],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[-11.913138 , -11.890651 , -11.718769 , ..., -0.6516266 ,
-0.11420345, -0.2889433 ],
[ -2.3279111 , -0.83327794, -0.38966238, ..., -0.18499136,
-0.31412745, -0.78526115],
[ -1.2105832 , -0.65170074, -0.30061066, ..., -0.5946727 ,
-0.5343251 , -0.26280165],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.108984 , -8.4880495 , -7.7428994 , ..., 3.0713623 ,
3.0152838 , 3.4003892 ],
[ -0.19017792, -0.29030776, -0.33207464, ..., -0.0172677 ,
-0.24603939, -0.9277673 ],
[ -0.6175966 , -0.8139999 , -1.1248744 , ..., -0.649323 ,
-0.6167655 , -0.5438595 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]]], dtype=float32)>,
3: <tf.Tensor: shape=(8, 267, 240), dtype=float32, numpy=
array([[[ -8.988267 , -8.965929 , -8.728106 , ..., 2.1489723 ,
2.334958 , 2.5260453 ],
[ -0.9514227 , -0.8912209 , -0.42839706, ..., -0.7747984 ,
-0.88131857, -1.1858144 ],
[ -1.6392798 , -1.8752912 , -1.3590232 , ..., -0.71193075,
-0.46426392, -0.79004574],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.117433 , -8.756327 , -8.3021965 , ..., 2.6770406 ,
2.9165506 , 2.619464 ],
[ -0.8334371 , -0.83099353, -0.7541424 , ..., -1.08567 ,
-0.3741293 , -0.14404964],
[ -1.1120342 , -1.2024816 , -1.1394764 , ..., -1.3766332 ,
-1.0821438 , -0.808341 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -8.535436 , -8.446434 , -8.433824 , ..., 1.7634034 ,
1.9151568 , 2.234317 ],
[ -0.73761785, -1.0498412 , -1.1566993 , ..., -1.9087753 ,
-1.4135947 , -0.43615532],
[ -3.1867118 , -1.3442845 , -0.6116022 , ..., -1.2935882 ,
-1.5357218 , -1.4339061 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
...,
[[ -8.752668 , -8.795702 , -9.234339 , ..., 1.4543443 ,
1.308625 , 1.8872037 ],
[ -0.8426125 , -0.5590738 , -0.64102423, ..., -0.2699995 ,
-0.34920692, -0.23712015],
[ -0.49331522, -1.201171 , -1.1223062 , ..., -1.4179659 ,
-0.8049512 , -0.8301816 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[ -9.640016 , -9.385439 , -9.126853 , ..., 2.6549459 ,
2.8241823 , 3.1816008 ],
[ -1.9176644 , -1.4502705 , -0.9663389 , ..., -0.6905961 ,
-0.6539016 , -0.5099206 ],
[ -1.4909953 , -1.6704174 , -1.1156216 , ..., -1.4263935 ,
-0.81988144, -1.1464152 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]],
[[-11.273719 , -10.959808 , -10.497703 , ..., 1.7455099 ,
1.9854834 , 2.464494 ],
[ -2.0011683 , -1.4769925 , -0.9517286 , ..., -0.25176716,
-0.5848155 , -0.5404253 ],
[ -1.4961203 , -1.4866172 , -1.359714 , ..., -1.1029644 ,
-1.4994144 , -0.89810133],
...,
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 0. ,
0. , 0. ]]], dtype=float32)>
}, PerReplica:{
0: <tf.Tensor: shape=(8, 19), dtype=int32, numpy=
array([[ 0, 500, 7, 1030, 2607, 29, 108, 2559, 4196, 4131, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 70, 75, 2098, 14, 1025, 228, 26, 1178, 8, 5,
3233, 4131, 0, 0, 0, 0, 0, 0],
[ 0, 12, 1849, 4117, 3926, 156, 64, 10, 932, 29, 108,
302, 4131, 0, 0, 0, 0, 0, 0],
[ 0, 1, 3220, 4196, 4130, 2585, 1846, 1, 418, 45, 10,
164, 4131, 0, 0, 0, 0, 0, 0],
[ 0, 1, 426, 56, 215, 1333, 173, 304, 4131, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 108, 2186, 290, 600, 22, 931, 266, 631, 56, 42,
2335, 4199, 4131, 0, 0, 0, 0, 0],
[ 0, 23, 2684, 121, 5, 88, 4185, 2864, 99, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 4, 486, 471, 12, 5, 69, 30, 1286, 26, 495,
397, 332, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
1: <tf.Tensor: shape=(8, 19), dtype=int32, numpy=
array([[ 0, 20, 701, 16, 2598, 442, 20, 502, 2628, 494, 1363,
1566, 0, 0, 0, 0, 0, 0, 0],
[ 0, 399, 26, 356, 52, 2941, 4131, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 1, 575, 1465, 4117, 3062, 4130, 25, 22, 2190, 2940,
4193, 4131, 0, 0, 0, 0, 0, 0],
[ 0, 9, 97, 3737, 17, 59, 3619, 474, 1846, 4196, 4131,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 84, 1332, 177, 3837, 4117, 939, 1316, 4200, 4131, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 3, 123, 2750, 4117, 3, 1462, 4117, 27, 5, 875,
4131, 0, 0, 0, 0, 0, 0, 0],
[ 0, 3019, 1525, 4007, 592, 664, 16, 289, 11, 51, 4028,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 2321, 4117, 25, 59, 1002, 18, 365, 150, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
2: <tf.Tensor: shape=(8, 19), dtype=int32, numpy=
array([[ 0, 143, 12, 108, 539, 7, 34, 67, 25, 2746, 4148,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 9, 2373, 195, 496, 190, 4, 3921, 74, 2677, 1078,
45, 1, 532, 19, 23, 140, 4131, 0],
[ 0, 23, 18, 4026, 456, 2204, 15, 17, 35, 831, 470,
524, 0, 0, 0, 0, 0, 0, 0],
[ 0, 5, 512, 2, 36, 18, 724, 21, 20, 688, 4131,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 78, 68, 36, 89, 20, 157, 1279, 1855, 1910, 611,
528, 747, 8, 139, 2423, 4200, 4131, 0],
[ 0, 3, 28, 12, 3, 582, 4131, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 92, 2640, 186, 4, 1235, 10, 1718, 459, 4131, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 614, 1674, 2151, 492, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
3: <tf.Tensor: shape=(8, 19), dtype=int32, numpy=
array([[ 0, 1044, 4130, 25, 1, 2312, 269, 1781, 4117, 21, 191,
4079, 4131, 0, 0, 0, 0, 0, 0],
[ 0, 35, 2797, 4117, 1290, 4196, 4130, 1517, 737, 12, 1411,
2873, 4182, 4131, 0, 0, 0, 0, 0],
[ 0, 49, 641, 1525, 3145, 15, 25, 23, 411, 1628, 403,
4131, 0, 0, 0, 0, 0, 0, 0],
[ 0, 9, 2824, 335, 646, 25, 13, 3178, 633, 2679, 29,
5, 916, 34, 1558, 181, 4182, 4131, 0],
[ 0, 92, 112, 4025, 4182, 4118, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 37, 273, 280, 1296, 19, 23, 140, 4131, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 26, 3288, 15, 493, 1239, 1424, 117, 37, 5, 517,
266, 138, 4131, 0, 0, 0, 0, 0],
[ 0, 932, 29, 1, 1383, 416, 4117, 12, 4, 1620, 1170,
1737, 420, 1138, 4117, 2, 5, 1785, 4131]], dtype=int32)>
}, PerReplica:{
0: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([267, 225, 193, 132, 140, 159, 105, 137], dtype=int32)>,
1: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([166, 126, 146, 132, 135, 180, 184, 163], dtype=int32)>,
2: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([137, 239, 157, 172, 201, 112, 127, 124], dtype=int32)>,
3: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([200, 174, 151, 185, 165, 176, 204, 209], dtype=int32)>
}, PerReplica:{
0: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([ 9, 12, 12, 12, 8, 13, 8, 12], dtype=int32)>,
1: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([11, 6, 12, 10, 9, 11, 10, 8], dtype=int32)>,
2: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([10, 17, 11, 10, 17, 6, 9, 4], dtype=int32)>,
3: <tf.Tensor: shape=(8,), dtype=int32, numpy=array([12, 13, 11, 17, 5, 8, 13, 18], dtype=int32)>
}, PerReplica:{
0: <tf.Tensor: shape=(8, 18), dtype=int32, numpy=
array([[ 500, 7, 1030, 2607, 29, 108, 2559, 4196, 4131, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 70, 75, 2098, 14, 1025, 228, 26, 1178, 8, 5, 3233,
4131, 0, 0, 0, 0, 0, 0],
[ 12, 1849, 4117, 3926, 156, 64, 10, 932, 29, 108, 302,
4131, 0, 0, 0, 0, 0, 0],
[ 1, 3220, 4196, 4130, 2585, 1846, 1, 418, 45, 10, 164,
4131, 0, 0, 0, 0, 0, 0],
[ 1, 426, 56, 215, 1333, 173, 304, 4131, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 108, 2186, 290, 600, 22, 931, 266, 631, 56, 42, 2335,
4199, 4131, 0, 0, 0, 0, 0],
[ 23, 2684, 121, 5, 88, 4185, 2864, 99, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 4, 486, 471, 12, 5, 69, 30, 1286, 26, 495, 397,
332, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
1: <tf.Tensor: shape=(8, 18), dtype=int32, numpy=
array([[ 20, 701, 16, 2598, 442, 20, 502, 2628, 494, 1363, 1566,
0, 0, 0, 0, 0, 0, 0],
[ 399, 26, 356, 52, 2941, 4131, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 1, 575, 1465, 4117, 3062, 4130, 25, 22, 2190, 2940, 4193,
4131, 0, 0, 0, 0, 0, 0],
[ 9, 97, 3737, 17, 59, 3619, 474, 1846, 4196, 4131, 0,
0, 0, 0, 0, 0, 0, 0],
[ 84, 1332, 177, 3837, 4117, 939, 1316, 4200, 4131, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 3, 123, 2750, 4117, 3, 1462, 4117, 27, 5, 875, 4131,
0, 0, 0, 0, 0, 0, 0],
[3019, 1525, 4007, 592, 664, 16, 289, 11, 51, 4028, 0,
0, 0, 0, 0, 0, 0, 0],
[2321, 4117, 25, 59, 1002, 18, 365, 150, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
2: <tf.Tensor: shape=(8, 18), dtype=int32, numpy=
array([[ 143, 12, 108, 539, 7, 34, 67, 25, 2746, 4148, 0,
0, 0, 0, 0, 0, 0, 0],
[ 9, 2373, 195, 496, 190, 4, 3921, 74, 2677, 1078, 45,
1, 532, 19, 23, 140, 4131, 0],
[ 23, 18, 4026, 456, 2204, 15, 17, 35, 831, 470, 524,
0, 0, 0, 0, 0, 0, 0],
[ 5, 512, 2, 36, 18, 724, 21, 20, 688, 4131, 0,
0, 0, 0, 0, 0, 0, 0],
[ 78, 68, 36, 89, 20, 157, 1279, 1855, 1910, 611, 528,
747, 8, 139, 2423, 4200, 4131, 0],
[ 3, 28, 12, 3, 582, 4131, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 92, 2640, 186, 4, 1235, 10, 1718, 459, 4131, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 614, 1674, 2151, 492, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
3: <tf.Tensor: shape=(8, 18), dtype=int32, numpy=
array([[1044, 4130, 25, 1, 2312, 269, 1781, 4117, 21, 191, 4079,
4131, 0, 0, 0, 0, 0, 0],
[ 35, 2797, 4117, 1290, 4196, 4130, 1517, 737, 12, 1411, 2873,
4182, 4131, 0, 0, 0, 0, 0],
[ 49, 641, 1525, 3145, 15, 25, 23, 411, 1628, 403, 4131,
0, 0, 0, 0, 0, 0, 0],
[ 9, 2824, 335, 646, 25, 13, 3178, 633, 2679, 29, 5,
916, 34, 1558, 181, 4182, 4131, 0],
[ 92, 112, 4025, 4182, 4118, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 37, 273, 280, 1296, 19, 23, 140, 4131, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[ 26, 3288, 15, 493, 1239, 1424, 117, 37, 5, 517, 266,
138, 4131, 0, 0, 0, 0, 0],
[ 932, 29, 1, 1383, 416, 4117, 12, 4, 1620, 1170, 1737,
420, 1138, 4117, 2, 5, 1785, 4131]], dtype=int32)>
}))
input_signature: (
[TensorSpec(shape=(None, None, 240), dtype=tf.float32, name=None), TensorSpec(shape=(None, None), dtype=tf.int32, name=None), TensorSpec(shape=(None,), dtype=tf.int32, name=None), TensorSpec(shape=(None,), dtype=tf.int32, name=None), TensorSpec(shape=(None, None), dtype=tf.int32, name=None)])
I can train the model use multi-gpus by add a decorator @tf.function refer this link https://github.com/tensorflow/tensorflow/issues/29911,and i also add line of “os.environ['CUDA_VISIBLE_DEVICES'] = "{your gpus}” in my code.
It's the multi-gpu training code what i modified,but the loss value from negative to nan after trained some batches.
