Albert Zeyer

> I would suggest to do something similar like `keep_over_epoch` but instead of returning a variable, return a placeholder. Any thoughts on this? This is another option. This is also...

Note that with our generalized self-attention implementation (see #391, and `nn.SelfAttention` in returnn-common), all the state handling is explicit, and this would also not really be a problem. You need...

How do you do the ONNX conversion? I assume you first create a TF graph, and then convert it somehow? How do you create the TF graph? Do you use...

> yes, i use `compile_tf_graph.py` for this and then call the tf2onnx tool on the resulting graph. What options? `--rec_step_by_step`?

I guess this can be closed. You did not answer the last question?

> I don't use `rec_step_by_step` option. Why not? This would automatically solve all your problems. This is exactly intended for this use case. We already use it for very similar...

I wonder, that is libcufft version 10. But there is CuFFT 12 now. Maybe we should try with a newer CuFFT version? **Edit** The libcufft filename versioning is confusing. It's...

Can you maybe try `tensorflow/tensorflow:2.15.0-gpu` as the base image? That comes with `libcufft-12-2` ([ref](https://github.com/tensorflow/tensorflow/blob/v2.15.0/tensorflow/tools/tf_sig_build_dockerfiles/devel.packages.txt)). I think that would be the file `libcufft.so.11.0.8.91` or so? In all references to this crash,...

Can you post the full log? Esp, what TF version is relevant here, but also the TF flags, etc.

So 4bfcc86c07095faacd6d5650347825c523a29a0f fixed it?