[Feature Request] Automatically log Tensorboard hyperparameters

[BlakeJC94]

trafficstars

I'm using ClearML with my PyTorch Lightning project, which logs hyperparameters in a TensorBoard format. To log hyperparameters I've used this workaround:

import pytorch_lighting as pl
from clearml import Task

from my_pytorch_code import MyTrainDataSet


class MyLightningModel(pl.module):
    def __init__(self, lr):
        self.lr = lr
        self.save_hyperparameters()
        ...

    def forward(self, x):
        ...

class MyDataModule(pl.LightningDataModule):
    def __init__(self, train_dataset, test_dataset, batch_size, num_workers):
        super().__init__()

        self.train_ds = train_dataset
        self.test_ds = test_dataset
        self.num_workers = num_workers
        self.batch_size = batch_size
        self.save_hyperparameters()

    def train_dataloader(self):
        return DataLoader(self.train_ds, batch_size=self.batch_size, num_workers=self.num_workers)

    def test_dataloader(self):
        return DataLoader(self.test_ds, batch_size=self.batch_size, num_workers=self.num_workers)


task = Task.init(...)

model = MyLightningModel(lr=1e-5)

datamodule = MyLightingDataModule(
    train_dataset=MyTrainDataSet(),
    batch_size=64,
)

# Workaraound: manually connect hparams to ClearML
task.connect(datamodule.hparams)
task.connect(model.hparams)

trainer = pl.Trainer()
trainer.fit(model, datamodule)

Is is possible to add functionality to ClearML to automatically check if model has a hparams attribute, and to connect hparams automatically if present?

[jkhenning]

Hi @BlakeJC94,

Usually with TB hparams are per-run parameters being used. With PyTorch Lightning we usually intercept the arguments passed to the argparser (in new versions the jsonargparse).

Kudos on the quick integration (literally one line per object), I would improve with section per object:

task.connect(datamodule.hparams, 'data')
task.connect(model.hparams, 'model')

Notice that when running with an agent you can change the logged values in the UI and it will take effect and change the values back on the object. Where would you suggest we have the auto-magic hook? is it save_hyperparameters?

[erezalg]

Hi @BlakeJC94, clearml 1.4.0 is out supporting autologging of TB hparams! :smile: Let us know if it works as expected!

[BlakeJC94]

Excellent, thanks for your help!