Albert Zeyer
Albert Zeyer
One option: E.g. `Data.feature_dim_axis` returns some default axis when it is unique (e.g. the only static dim), and otherwise `raise Ambiguous` or so. Same for `time_dim_axis`. It would be disallowed...
> What we need in any case here is a list of layers and other util functions which make use of these special axes. Maybe a wiki page should be...
I still wonder whether there are still valid use cases for this. Or rather, when thinking about how usual code would look like, it is very common that there is...
Maybe it makes also sense for some intermediate solution. I think it's fine to treat the batch dim special anyway. And the feature dim can maybe also be kept special...
We could also rely on the `DimensionTag.kind` and allow `"B"`, `"T"`, `"F"` exactly when it is unique according to the dim tags.
For reference, with `debug_add_check_numerics_ops = True`: ``` InvalidArgumentError: optimize/gradients/lstm0_bw/rec/NativeLstm2_grad/GradOfNativeLstm2:0 : Tensor had NaN values [[node add_check_numerics_ops/optimize/gradients/lstm0_bw/rec/NativeLstm2_grad/GradOfNativeLstm2_check_numerics (defined at /work/tools/asr/python/3.8.0_tf_1.15-generic+cuda10.1/lib/python3.8/site-packages/tensorflow_core/python/framework/ops.py:1748) ]] ``` Or: ``` InvalidArgumentError: optimize/gradients/output/rec/att/TensorArrayStack/TensorArrayGatherV3_grad/TensorArrayGrad/TensorArrayGradV3:1 : Tensor had NaN values...
I just noticed that when I run `python3 tests/test_TFEngine.py` (i.e. all the tests in there), sometimes I get inf/nan. This is with TF 1.15.2, CUDA 10.0, GTX 2070. It is...
`python3 tests/test_TFEngine.py test_rec_subnet_train_t3d_simple` can reproduce it (sometimes, actually quite rarely, maybe in 2% of the cases, so run in maybe 100 times to get the error), and that is even...
Some success! I have created a TF standalone test case [here](https://github.com/albertz/playground/blob/08ece8b7a18b4568ef3c48874d944d57be1eb9e0/tf-test-tensorarray-bug.py) ([latest version](https://github.com/albertz/playground/blob/master/tf-test-tensorarray-bug.py)). This test case has some interesting behavior: * TF 1.14.0, CUDA 10.1, Python 3.7, i6: Tensor had...
Some update: Whenever `CheckNumerics` catches something `TensorArray` related, this might be a false positive. The TensorArray uses a dummy "flow" tensor (for gradient management). The value of this flow tensor...