Lucas Weber
Lucas Weber
I had similar generations for multi-GPU runs. Setting random seeds made them coherent for me.
@hzy312 I am having the same problem. The script gets stuck on feeding the input into the first layer. Did you find a solution to this problem?
@andrewmlu I don't think it has to do with the amount of vRAM, but something else related to the type of GPU (maybe the drivers?). For me, the problem was...
I have solved the issue (for me). As far as I understand, has to do p2p communication of the GPUs via NCCL. One can deactivate p2p via setting environmental variables...
@carlos-gemmell I ran into a similar issue and suspect it has to do with available RAM. I will report if it helps me to resolve the problem EDIT: Nevermind
I have the same issue. Single-node runs are fine, while multi-node runs are gibberish
@YuzhongHuangCS Thanks so much, this works like a charm!
> Thanks for the PR! We're glad that lm-eval-harness was useful to you during development :)) > > A few things: > > * Would you be able to sign...
I looked a bit into this it seems to me as if it is easy to modify the existing metrics `acc` and `acc_norm` (e.g. by adding a custom aggregation function,...
> @LucWeber what would you need for the `acc_irt` metric? In addition to the IRT model, would the inputs be the same as `acc`? Hey @lintangsutawika, thanks for coming back...