genienlp
genienlp copied to clipboard
Support for non-almond datasets
Through several rounds of changes we've made so far, the generic datasets (originally part of decaNLP) cannot be loaded properly in genienlp anymore and should be fixed. Also, many of those datasets (e.g. iwslt, sst, squad, imdb) are now part of datasets library. We should remove those obsolete implementations and associated metrics, and import them from datasets instead. We have added support to load new datasets from HF datasets. Check out https://github.com/stanford-oval/genienlp/blob/master/genienlp/tasks/hf_dataset.py for the implementation.