Albert Zeyer

More examples: ``` output/exp_fs_base/conformer_pre10_d384_h6_blstmf2_oldspecaug4a_oldtwarp_attdrop01_aux48/recog_results_per_epoch/150 {"hub5e_00": 23.4, "hub5e_01": 15.2, "rt03s": 21.2} output/exp_fs_base/conformer_pre10_d384_h6_blstmf2_oldspecaug4a_oldtwarp_attdrop01_aux4812/recog_results_per_epoch/150 {"hub5e_00": 22.9, "hub5e_01": 16.0, "rt03s": 21.3} ``` And: ``` output/exp_fs_base/conformer_pre10_d384_h6_blstmf2_specaug_attdrop01_posdrop01_aux48/recog_results_per_epoch/150 {"hub5e_00": 19.5, "hub5e_01": 15.7, "rt03s": 19.2} output/exp_fs_base/conformer_pre10_d384_h6_blstmf2_specaug_attdrop01_posdrop01_aux4812/recog_results_per_epoch/150 {"hub5e_00": 20.9,...

> Correction: The TTS experiments were with new seeds set on purpose, so we can exclude that here. So you say with same seed, they are very deterministic, or you...

I also use Conformer here, instead of BLSTM as I did in similar earlier determinism experiments. Maybe Conformer also leads to more non-determinism?

I recently observed some potential non-determinism in gradient accumulation and maybe other things which make use of the global train step (https://github.com/rwth-i6/returnn/issues/1205). The PR https://github.com/rwth-i6/returnn/pull/1206/ is supposed to fix that...

Maybe this is also related to returnn-common in some way. These are pure returnn-common setups, i.e. this is also a new SpecAugment implementation, etc.

A starting point would be to check and update our `get_non_deterministic_ops_from_graph` function. I think this is not up-to-date anymore.

Ah yea, many things such as the code itself and other flags are part of the hash for these cache files, but this particular flag was not part, thus you...

We don't really want to have any other dependencies, except of GCC itself. I also don't think that ccache would really give us any benefit over what we currently have.

Well ccache is still another separate dependency and usually not installed by default, but we really want to avoid further dependencies and we also want that RETURNN basically "just works"...

Note, I think we can write some scripts: - One which automatically performs changes, moving type spec from docstring to type annotation, fixing wrong indents. - One which checks if...