gritlm icon indicating copy to clipboard operation
gritlm copied to clipboard

Published 20 hours ago • verified publisher icon ContextualAI

Dose your evaluation code support data parallelism?

Open selmiss opened this issue 1 year ago • 3 comments

Dose your evaluation code support data parallelism? I couldn't find any module to mteb evaluation that use DP or DDP

selmiss avatar Mar 07 '24 04:03 selmiss

Yes it automatically uses DP via this line of code https://github.com/ContextualAI/gritlm/blob/b89fdefa18731f1aa1d6111c3849c1e4c811b9d6/gritlm/gritlm.py#L75

Muennighoff avatar Mar 07 '24 15:03 Muennighoff

Thank you. And by the code we can only use DP or PP, can't mix using these two?

selmiss avatar Mar 08 '24 05:03 selmiss

Thank you. And by the code we can only use DP or PP, can't mix using these two?

Yeah only one or the other for mteb evaluation; I'm not sure if transformers supports mixing them

Muennighoff avatar Mar 08 '24 05:03 Muennighoff