gpt-neox icon indicating copy to clipboard operation
gpt-neox copied to clipboard

Published 20 hours ago • verified publisher icon EleutherAI

add merge script

Open Mistobaan opened this issue 3 years ago • 4 comments

re-based version of https://github.com/EleutherAI/gpt-neox/pull/466

Tested only on 20B

Mistobaan avatar Feb 13 '22 00:02 Mistobaan

@Mistobaan in the referenced PR, it is found that the merge reduces performance. Is this still the case in your verison?

StellaAthena avatar Feb 16 '22 01:02 StellaAthena

Not sure, which benchmarks are we running, and on what hardware?

Mistobaan avatar Feb 16 '22 02:02 Mistobaan

@Mistobaan https://github.com/EleutherAI/gpt-neox/pull/466#issuecomment-997517986

EricHallahan avatar Feb 16 '22 02:02 EricHallahan

oh, I see, you mean the accuracy of the model is not the same when the weights are merged. Yes this is still the case with this PR.

Mistobaan avatar Feb 16 '22 19:02 Mistobaan