model-remediation icon indicating copy to clipboard operation
model-remediation copied to clipboard

TF Dataset to TF Examples List in FDW Utils: Implementation is slow and consumes excessive RAM

Open CLSchmitz opened this issue 2 years ago • 0 comments

The current implementation of TF Dataset to TF Examples List uses a lot of RAM and runs very slowly on large datasets.

In this colab, the CelebA dataset, as processed by the fairness indicators tutorial, can't be processed on a default hosted runtime. After 18-20 minutes, the runtime crashes because RAM requirements are exceeded.

Because of all the preprocessing required, it's not easy to create a smaller reproducible example.

CLSchmitz avatar Aug 14 '22 17:08 CLSchmitz