Tadej Svetina

More context: the memory leak is most likely due to caching, if I remove `cache_dir` argument, RAM usage stays low and constant

@tmbdev pinging just in case you missed this

Awesome, happy to hear that.

I propose to use, when possible, huggingface datasets. They are extremely easy to use, and very performant too.

I agree with this - how come there is such a huge discrepancy between the method used in the paper and the one in the repo? I would appreciate some...

I can, but I don't want to :)

Well, what led me to write this FR is that at first I missed the `-w 0` option[^1] for `base64`, and wasted ~15 minutes of my time on this -...

@tmbdev Do you still plan to work on this (and related) PRs?

@VitalyFedyunin Mind taking a look at this PR?

@FateScript what is the status of this PR?