Evaluate on multiple gpus

[ariecattan]

Thanks for making easily available all metrics for summarization!

Some metrics (e.g QAGS, QAEval) are a bit heavy to run and it'd be great if we can parallelize the evaluation of multiple devices.

[danieldeutsch]

Thanks for the suggestion! I definitely agree this would be useful, but I don't have the bandwidth right now to implement that. I think the change is non-trivial.

What I've done in these situations is split the data into shards, run each shard on its own process, then combined the results.

Sorry that I cannot help right now!

[danieldeutsch]

Hi Arie,

If you are still wanting parallel evaluations, I've been working on a new library in which a bunch of models/metrics are Dockerized, which makes them way easier to use if you have access to a machine with Docker. I just pushed an abstraction that allows for easily running the models/metrics in parallel across multiple devices. Hopefully you find it useful!

Dan

[ariecattan]

Hi Dan,

Thanks for the pointer, your new repo is very cool, I'll start using it! Yes, this is very useful that we can easily run any model on multiple devices.

Arie