Albert Zeyer

Docstring format and type annotations

2

Now that we only need to target Python >=3.7 (#487), we can make use of annotations, and also use forward declarations (no need to stringify types), although this needs: ```python...

HuggingFace datasets wrapper

4

We should be able to use [HuggingFace datasets](https://huggingface.co/docs/datasets/index) directly in RETURNN. I guess the most canonical way would be to write a RETURNN `Dataset` for this. Maybe derived from `CachedDataset2`....

Replace PyCharm code inspections by faster and easier tool?

11

One problem with the PyCharm code inspections is that they are quite slow to run, and delay the CI tests. (Related was the earlier discussion on `pycharm-inspect` here: #328) Also,...

Suppose some logic like: ```python with nn.Cond(nn.train_flag()) as cond: cond.true, out_spatial_dim = nn.resize(x, axis=in_spatial_dim, factor=0.9) cond.false = x y = cond.result ``` The output of the `CondLayer` has different spatial...

Horovod training MPI_Allreduce MPI_ERR_INTERN: internal error, with restart_after_num_net_reinit

3

It already trained 10 subepochs, and then `restart_after_num_net_reinit` became effective, and then I got this error: ``` ... reinit network too often, 1 times after 10 training epochs, restart ......

Currently our implemented constraints are: - L2 on weights (`L2` option on a layer) - Some exotic things on activations (`darc1`, `spatial_smoothing`) We already have the possibility to decouple the...

OpCodeCompiler/TFNativeUtilCompiler, better C++ standard version detection, use get_compile_flags

9

RETURNN now has TF 2.10 support and we also use that for CI (#1160). TF 2.10 by default uses C++17 now (https://github.com/tensorflow/tensorflow/commit/8ea5ed0c392b329a3e0481a3f1f7b0ca86821b84) and thus this is also used for the...

Support dim tag with static shape and dyn_size_ext

4

`dyn_size_ext` means that it has different sequence lengths per batch entry (or whatever `dyn_size_ext` shape is). Currently this implies a dynamic shape (`None` dimension), and vice versa, a dynamic shape...

It would define a number of outputs by your model. Just in the same format as `extern_data`. This could be used by tools like the compile-TF-graph script. It can check...

This depends on: - https://github.com/rwth-i6/returnn/issues/1161 - https://github.com/rwth-i6/returnn/issues/1139 - (Maybe optional: https://github.com/rwth-i6/returnn/issues/296 or https://github.com/rwth-i6/returnn/issues/1145) It might need some further changes which I'm currently not aware of, but I guess we will...