Maanu Grover comments

Results 23 comments of


                                            Maanu Grover

Add flag for creating scaling.json in package checker

Should we add any info in the `package_checker/README.md` about this flag?

package_checker creates file scaling.json in result directory

I think this is because of #271 . You can certainly get rid of these files before creating submission with something like: `find -type f -name 'scaling.json' -delete` . We...

llama2 training hangs when pp_size > 1

Hi @PurvangL , I see you've closed this issue, were you able to resolve? I haven't had time to reproduce this issue with SFT, but I've encountered long init times...

setting sync_dist to true for validation metrics

Another note, from my experience, `sync_dist=True` is tricky with parallelisms, especially pipeline parallelisms. If some metrics are non-existent on some ranks (eg val_loss is only on last pp stage, so...

The phrase "need to want to" is grammatically incorrect

rerouted this change to `model_provider.py` and included in commit e002b5c23c0f8681bbf42df1c3eb07ec27ff5446

slurm Multi-machine and multi-GPU training

Hi @yangzhipeng1108, looking at the error, > ``` > 2: File "/usr/local/lib/python3.10/dist-packages/lightning_fabric/plugins/environments/torchelastic.py", line 63, in world_size > 2: return int(os.environ["WORLD_SIZE"]) > 2: File "/usr/lib/python3.10/os.py", line 680, in getitem > 2:...

Maanu Grover