genie-toolkit icon indicating copy to clipboard operation
genie-toolkit copied to clipboard

Published 20 hours ago • verified publisher icon stanford-oval

Augmentation is memory hungry

Open sileix opened this issue 2 years ago • 0 comments

Currently, the augmentation step will load all parameter values to memory, which could be an issue for some cases like QA over Wikidata. We could either use mmap or do a pre-sampling over the dataset.

sileix avatar Sep 12 '21 04:09 sileix