Leandro von Werra

To clarify, I don't think it loops indefinitely but the `paj.read_json` gets stuck after the first try. That's why I think it could be an issue with a lock somewhere....

I agree that it is not super rigorous, but since we run the commands sequentially in the for-loop shouldn't that prevent systematic issues? Also wouldn't explain why without the `lfs_log_progress`...

No it did not - it happened overnight and it started around 11pm and I noticed it failed at 9am. So using `git lfs` worked and was **much** faster. In...

Just the standard ones: ```bash git add git push ``` Note that I only uploaded the dataset once so there could have been a temporary bandwidth issue on GCP or...

I am also in favour of this and at the same time pushing the `Repository` class for more sophisticated use-cases where a little more code is required but much more...

We could update ROUGE similiar to BLEU (#19) where optionally a tokenizer can be passed. The `RougeScorer` accepts a `tokenizer` keyword argument. cc @sashavor

Also listed a few libraries in #11 e.g. NetworkX for graph metrics.

Thanks for your feedback! @lhoestq I reworked the logic based on your feedback. Is that what you had in mind?

>Do you know how users can get some docstrings about the config parameters ? This would be useful to document IMO (can be done in a subsequent PR) The can...

@sashavor I also refactored `toxicity` a bit to better fit with the other modules. Let me know if you agree with the changes (as well as all the others :))!...