Albert Zeyer
Albert Zeyer
Many of the jobs `hash` functions modify `kwargs` inplace. E.g.: ```python class CountNgramsJob(Job): ... @classmethod def hash(cls, kwargs): """delete the queue requirements from the hashing""" del kwargs["mem_rqmt"] del kwargs["cpu_rqmt"] del...
Currently I run `setup_job_symlinks()` (code below) at the end of some of my Sis scripts (Sis config), which will go through the Sis job graph, look for matching jobs, and...
I just want to raise this here. Also including the README. A lot of parts are still TF specific. And many of those even don't mention that. PyTorch relevant documentation...
Again a crash. It ultimately failed with this (after retrying a few times): ``` OSError: [Errno 28] No space left on device ``` Log: ``` FileCache: Copy file /rwthfs/rz/cluster/home/az668407/setups/2025-08-aed-large/work/i6_core/datasets/huggingface/TransformAndMapHuggingFaceDatasetJob.F xPUVJtw1EeN/output/dataset/train/data-00405-of-00848.arrow...
CI run log [tf-tests (3.8, 2.10.0, TEST=TFNetworkLayer)](https://github.com/rwth-i6/returnn/actions/runs/18276719792/job/52030410528?pr=1774#logs). ``` Python env: python is /opt/hostedtoolcache/Python/3.8.18/x64/bin/python Python 3.8.18 NumPy: 1.24.4 TensorFlow: v2.10.0-rc3-6-g359c3cdfc5f 2.10.0 /home/runner/.local/lib/python3.8/site-packages/tensorflow/__init__.py ``` Relevant log: ``` ___________________________ test_ConvLayer_empty_out ___________________________ Traceback (most...
_Originally posted in [#1257](https://github.com/rwth-i6/returnn/issues/1257#issuecomment-3185864489), on the `HuggingFaceDataset`, but actually, the discussion applies just the same for most of our datasets, e.g. `HDFDataset`, `OggZipDataset`, etc_ > [We] do random access here...
It is being called (after `handle_cached_files_in_config`, after we created a dataset with cached files). But then, the `_file_cache` attribute is never used anywhere, as far as I can see. This...
See my recently added `test_DistributeFilesDataset_sharding`. I was expecting that `global_seq_idx == len(hdf_files) * num_seqs // distrib_size` in the end. But this is not the case. When looking at `DistributeFilesDataset.init_seq_order`, I...
Currently, in many of our setups, we make use of a so-called "devtrain" dataset, which is just a subset of the train dataset, which is then used for evaluation (to...
With `MultiProcDataset` and `sharding_method = "seq_order"` (default), the seq order shuffling will be done by the first worker (sub dataset), and then we evenly split it up over the workers...