Albert Zeyer

Btw, as we work in master, please make sure the tests are passing. There is no PyTorch test at all yet, so the only test relevant for us currently is...

For the main train loop, see the [PyTorch quickstart tutorial](https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html) (slightly simplified, adapted): ```python device = "cuda" if torch.cuda.is_available() else "cpu" class NeuralNetwork(nn.Module): ... model = NeuralNetwork().to(device) loss_fn = nn.CrossEntropyLoss()...

So, in the RETURNN `Engine`, we would also have the same to loop over the batches: ```python for batch, (X, y) in enumerate(dataloader): X, y = X.to(device), y.to(device) ``` We...

No, the `IterableDataset` is supposed to return individual sequences, not batches. I think the `DataLoader` is supposed to do that. Yes, I have read somewhere that it is a bit...

> it looks to me that having a separate loss function in the training loop is not strictly necessary. I never said that? See my suggestion on how to do...

Btw, I don't like having PRs in this early phase of development. PRs just slow it down. I think we can all directly work in the master. PRs are there...

In general, I would trust the Fairseq developers more, esp w.r.t. how to use PyTorch in the best way. So, let's follow how they do it.

When wrapping the RETURNN dataset, I don't think any of these tutorials can be applied. You can just use the RETURNN Dataset API, nothing else. The Torch map-based dataset unfortunately...

I moved the `init_seq_order` into the `DatasetWrapper.__iter__`. I think this is cleaner.

`python3 rnn.py demos/demo-torch.config` runs now. I.e. it constructs the net, calcs and optimizes the loss. Many things are missing. The model is always random in each epoch. No storing/saving implemented...