Sergey Serebryakov

Add Kornia to benchmark

3

Added `kornia` library to benchmark suite, with a few of the augmentations. This was requested in #527. Kornia operates on PyTorch Tensors, so I introduced a method to read images...

Petastorm sharding + Distributed PyTorch

14

**Problem:** I would like to train a PyTorch model on a Parquet dataset in a distributed (multi-GPU, multi-machine) setup, for a fixed number of epochs. For this, I need to...

Fix GCSFS walk() method

6

Resolves #558

Suspected memory leak with `reader_pool_type='process'`

1

**Problem:** When running distributed Tensorflow training with Petastorm and `reader_pool_type='process'`, I've found that worker machines eventually run out of RAM. This is how I create a reader: ``` make_reader( ......

Also possible to use utility functions from Guava/Apache Commons. Also possible to split the functionality into ReadingParser/WritingParser or something along these lines. Relying on filesystem to handle concurrent file access.