Improvements to multinode papers100m default hyperparams and adding eval on all ranks

[puririshi98]

using main branch of PyG (16 GraceHopper nodes): Val Acc: 0.4546 Test Acc: 0.3770

using PR branch (2 GraceHopper nodes due to availability): Validation Accuracy: 51.1759% Test Accuracy: 44.5692%

PR ready

[puririshi98]

@akihironitta @rusty1s anything needed to get this merged?

[puririshi98]

@rusty1s lmk if anything else needed to merge