lightning-hydra-template icon indicating copy to clipboard operation
lightning-hydra-template copied to clipboard

Fixed seeding before instantiation of mlflow logger causes identical run names

Open ChristianHinge opened this issue 1 year ago • 0 comments

MLFlow generates random run names using the random module. Since random.seed is set before instantiation of the mlflow logger, the same run name is always generated.

https://stackoverflow.com/questions/76053273/how-can-i-set-the-seed-that-mlflow-uses-to-randomly-generate-run-names

My current fix is to instantiate the loggers before the line: L.seed_everything(cfg.seed, workers=True)

and forcing the setup of the logger during instatiation by logging some dummy variable:

...
    for _, lg_conf in logger_cfg.items():
        if isinstance(lg_conf, DictConfig) and "_target_" in lg_conf:
            log.info(f"Instantiating logger <{lg_conf._target_}>")
            _logger = hydra.utils.instantiate(lg_conf)
            logger.append(_logger)
            _logger.log_metrics({"dummy": 0})

    return logger

If the above is not added, the setup of the logger (which includes generating a random name), only happens at log_hyperparameters, and thus the seed would already be set and the run name would again be identical.

This feels like a hacky solution. Do you have any suggestions on a propper way to address the problem?

ChristianHinge avatar Oct 17 '23 09:10 ChristianHinge