composer icon indicating copy to clipboard operation
composer copied to clipboard

Refactor logging of `rank_zero_seed`

Open abhi-mosaic opened this issue 2 years ago • 2 comments

This line is logging the rank_zero_seed as if it is time series data, but it's a static value that is only configured once. It looks very out of place in logger backends like WandB. I think rank_zero_seed should be stored in config somewhere but not logged as time series.

This can be done by either promoting it to the TrainerHparams, or creating a new update_config() function that LoggerDestinations have to implement.

Also a UX bug -- if each rank is logging in distributed training, then the seed stored in each rank's config is now incorrect because Trainer has overridden it to be rank_zero_seed + rank. Are there any other codepaths we are overriding values in the config?

https://github.com/mosaicml/composer/blob/204129b810ce71d4089781b6ee2bef9f0b2703f1/composer/trainer/trainer.py#L873

abhi-mosaic avatar Apr 20 '22 20:04 abhi-mosaic

This is what WandB looks like right now: Screen Shot 2022-04-20 at 1 44 58 PM

abhi-mosaic avatar Apr 20 '22 20:04 abhi-mosaic

I like the idea of implementing update_config in the logger destinations, as different loggers may want to store config differently.

ravi-mosaicml avatar Apr 28 '22 01:04 ravi-mosaicml

Closed by PR #1416

eracah avatar Aug 31 '22 00:08 eracah