Albert Zeyer

In this current state, I don't use `ExternData`. Probably we want to introduce this. Not sure. But for getting a first proof-of-concept, we maybe also don't need it yet.

Current state: - PR #1137, done. - We discussed that it is probably simpler for everyone to not open PRs but directly push to master, as long as the code...

> For inference, we want to export to ONNX. It's a bit unclear how that looks like for the user. We might use `extern_data` to define the inputs, and have...

Note, I was thinking a bit about how to go forward later with the PyTorch backend of RETURNN, and whether to support the net dict, or how else we should...

I heard good things about PyTorch-lightning. One important feature is that it easily supports multi-GPU training, where you don't really need to change anything on the user code. As we...

> For now I only calculate the "score" and not the "error" of the eval sets as that's what the `train_step` function calculates at the moment. This is wrong. `train_step`...

> I just noticed that another thing still missing is keeping the optimizer state. Both between epochs - currently we recreate it in every epoch Yes this should be created...

I found some code [here](https://github.com/sermanet/OverFeat/blob/master/API/python/overfeatmodule.cpp).

Ping. What about it?

Note, I'm now using this extremely ugly hacky monkey patch: ```python orig_get_downloader = tfds.download.downloader.get_downloader def _patched_get_downloader(*args, **kwargs): kwargs.setdefault("max_simultaneous_downloads", 1) return orig_get_downloader(*args, **kwargs) tfds.download.downloader.get_downloader = _patched_get_downloader ``` This seems to work....