CUDA_ERROR_ILLEGAL_ADDRESS, CPU support, TensorFlow bindings, some other changes

[albertz]

Hey!

Thank you for this implementation!

I just wanted to say that we use your implementation and have done some changes, extensions and fixes to it. Unfortunately I don't really have time currently to prepare PRs for the individual changes, at least right now, and we anyway probably stay within our fork (but then I also don't expect that we will really do much further changes now). However, I still wanted to mention that we have done these changes, which are partly probably also interesting to you.

Our fork: https://github.com/rwth-i6/warp-rna

For example, we got a CUDA_ERROR_ILLEGAL_ADDRESS error, due to illegal memory access, reported here: https://github.com/rwth-i6/returnn/issues/1124. I think this was caused when you have cases where S > T. We fixed that now.

Further, we also implemented CPU support, in a somewhat hacky way, by wrapping up the CUDA code with some clever makros, which make it runnable on CPU without CUDA. See here to get an idea how this works.

And we also added TensorFlow bindings.

[1ytic]

Hey, Albert! I'm curious why you chose this implementation? Because it is a rare implementation of RNA or because of CUDA warps threads? In any case, I'm glad to see you use this project. I built this mostly as a toy example and do not use it anymore.

[albertz]

Do you know a better RNA implementation? I don't really remember anymore why we took this one.