Albert Zeyer
Albert Zeyer
In PyTorch distributed training, I get: ``` File "/rwthfs/rz/cluster/home/az668407/setups/combined/2021-05-31/tools/returnn/returnn/torch/engine.py", line 198, in Engine.init_train_from_config line: self._ddp_pt_model = self._torch_distributed_class( self._pt_model, device_ids=get_device_ids(), **self._torch_distributed_options ) File "/rwthfs/rz/cluster/work/az668407/py-envs/py3.10-torch2.1/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 795, in DistributedDataParallel.__init__ line: _verify_param_shape_across_processes(self.process_group, parameters)...
Our TF-layers Nadam optimizer is basically the same as Adam except that we use `use_nesterov=True` for `training_ops.apply_adam`. It is based on TF 1.15 [tensorflow/contrib/opt/python/training/nadam_optimizer.py](https://github.com/tensorflow/tensorflow/blob/v1.15.5/tensorflow/contrib/opt/python/training/nadam_optimizer.py). So it also has the same...
`rf.Parameter.weight_decay` is ignored in the PyTorch engine.
Pro `num_workers=1`: * It should always be faster, usually fix low computation time (#1383, #1435). * It should just work. Contra, i.e. leave the current default, `num_workers=0`: * It is...
The extra nets should set the new control flow context. It seems no code really depends on this though? So far everything worked fine. Note that the RF with TF-layers...
``` /u/zeyer/setups/combined/2021-05-31/tools/returnn/returnn/torch/frontend/_backend.py:469: UserWarning: Specified kernel cache directory could not be created! This disables kernel caching. Specified directory is /var//tmp/.unburden-zeyer/cache/torch/kernels. This warning will appear only once per process. (Triggered internally at...
I think we never officially stated the minimum PyTorch version we require for RETURNN (or did we? where?). I remember that 1.13 or so was mentioned (or assumed) in some...
[Haste: a fast, simple, and open RNN library](https://github.com/lmnt-com/haste). Could be both for TensorFlow and PyTorch. Put it under `extern/`, do it in a similar way like `blocksparse`, `WarpRna`, `HawkAaronWarpTransducer` or...
File `IpcFabricConfigClient.cpp`. ``` $ gdb --args python3 -m pytest tests/test_rf_array.py GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1 ... Reading symbols from python3... (gdb) r Starting program: /work/tools/users/zeyer/linuxbrew/bin/python3 -m pytest tests/test_rf_array.py [Thread debugging...
Via #487, we are now supporting Python >=3.7. Do we want to bump this and require Python >=3.8 or later? See also the Python feature list in #487. Similarly, Python...