Garrett Goon

I can't seem to comment on the images: in docs/assets/images/arch09.png, can we also include `DeepSpeedTrial`? Would require some graphic re-sizing for legibility.

First, I would expect to first pass a space-separated list of files and directories as a single `--context` arg, so `--context d1 f1` rather than `--context d1 --context f1`. >...

> Do your answers to 1. and 2. seem to indicate that `--context .` means different things based on if other `--context` options appear on the command line? No, I...

> To me it would be very confusing to have `--context f1 --context f2` to result in `f1` and `f2` in `workdir/`, but for `--context d1 --context d2` to not...

I think both the `--context .` and `--context d1` both have the ugly ducking behavior that we are maybe stuck with.

CC @jingxu10 @tye1, thank you!

Thank you @YuningQiu , greatly appreciated!

Hi @YuningQiu , thank you for the very detailed response! I have a few follow-ups. > By default, the initiation of the second allreduce is implicitly delayed until the first...

And for more color, all of the above seems consistent with what I have seen from the pytorch profiler. These are traces of a very similar workload where I attempted...

Hi @hueyline have you been able to reproduce this?