List of dictionary is not correctly displayed on HYPER PARAMETER Tab.

[S-aiueo32]

Hi, I'm building projects on the versions as below:

• Python: 3.8.6
• ClearML: 0.17.2
• Trains-Server: 0.16.1

The hyperparameters are provided as dict-like objects to configure the training settings through YAML or so. Basically, this style has pairs of variable names and parameters as key-value pairs. See the networks section in the following example. That has the variable name net with the parameters. It does not cause any problems.

config = {
    'networks': {
        'net': {
            '_target_': 'torchvision.models.resnet18',
            'pretrained': False
        }
    },
    ...
}

On the other hand, common frameworks like PyTorch Lightning require that optimizers are ordered according to the optimization order, so we need to feed the list of parameter dictionaries:

config = {
    ...
    'optimizers': [
        {'_target_': 'torch.optim.Adam', 'lr': 1e-3},
        {'_target_': 'torch.optim.Adam', 'lr': 1e-4}
    ]
}

task = Task.init(
    project_name='hoo',
    task_name='bar',
    auto_connect_arg_parser=False
)
task.connect(config, 'config')

The above optimizers config will be displayed like:

I understand it is expected behavior but I suggest these configs displayed as below to compare hyperparameters.

optimizers.0._target_ | torch.optim.Adam
optimizers.0.lr       | 1e-3
optimizers.1._target_ | torch.optim.Adam
optimizers.1.lr       | 1e-4

Thanks!

[bmartinn]

Hi @S-aiueo32 If you have a complex dictionary I think the best way to store/edit it would be with task.connect_configuration. For example:

config = {
    'optimizers': [
        {'_target_': 'torch.optim.Adam', 'lr': 1e-3},
        {'_target_': 'torch.optim.Adam', 'lr': 1e-4}
    ]
}
task.connect_configuration(config, name='config', description='Network configuration object')

Then in the web UI under "Configuration" -> "Configuration Object" you will have:

Wouldn't it be better ?

A few thoughts: We could definitely flatten lists like we do with dictionaries, the syntax looks good (HoCON does the same thing), but it feels a bit awkward, no? Since this is actually a configuration file (of a sort), it might be easier to view/edit in text, then translated back to nested dictionaries / lists in code, like we do with "connect_configuration"... WDYT?

BTW: Any reason not to connect the argparse? If I remember correctly lighting by default puts all the configuration there, making it quite easy to track and change from the UI. Am I missing something ?

[S-aiueo32]

Thank you for replying. Honestly, I use Hydra and Pytorch Lightning for my project so the topics around auto_connect_arg_parser are out of scope here. In addition, I set the Hydra integration off for later-mentioned reasons.

task = Task.init(
    ...
    auto_connect_arg_parser=False,
    auto_connect_frameworks={'hydra': False},
)

There are some reasons why I don't use connect_configuration:

1. Tab placement in "Compare Experiments" We can see configurations and hyperparameters at the same time for a single experiment. We can't, however, see them on "Compare Experiments" and we have to deeply dig into "Details" > "Configuration". I think it is not intuitive especially for collaboration with other teammates who has the other configuration style like Argparse. I'm sure that they will first see the "Hyperparameter" tab so I use connect.

2. Difficulty to compare We can certainly compare configurations on the "Compare Experiments" page, but there is difficulty to find differences in configurations. For example, comparing the following dictionaries will be collapsed because all lines of B under _target_ will be highlighted.

# A
config = {
    'networks': {
        'net': {
            '_target_': 'torchvision.models.resnet18',
            'pretrained': False
        }
    },
    'optimizers': [
        {'_target_': 'torch.optim.Adam', 'lr': 1e-3},
    ]
}

# B
config = {
    'networks': {
        'net': {
            '_target_': 'torchvision.models.resnet18',
        }
    },
    'optimizers': [
        {'_target_': 'torch.optim.Adam', 'lr': 1e-4},
    ]
}

BTW, I also think the flatten-dicts is a little bit awkward. Ideally, I want configurations and hyperparameters to be integrated like ReDoc (see "Request samples"), which can align the keys and keeps the hierarchy.

[bmartinn]

Tab placement in "Compare Experiments" We can see configurations and hyperparameters at the same time for a single experiment. We can't, however, see them on "Compare Experiments" and we have to deeply dig into "Details" > "Configuration". I think it is not intuitive especially for collaboration with other teammates who has the other configuration style like Argparse. I'm sure that they will first see the "Hyperparameter" tab so I use connect.

I think this is a very good point, it makes sense to have them next to the hyper parameters, but I think that since that hyper-parameters are key/value tables (or in displayed on a parallel coordinates graph), this is not really feasible. Maybe we should have a separate tab for configuration objects, what do you think?

2. I see your point, but I think only the 'pretrained': False and the {'_target_': 'torch.optim.Adam', 'lr': 1e-4}, should be different. The comparison itself is quite basic as it is a line by line, by I suspect that there is a bug in the comparison algorithm. I think that with a better text-based comprison this section will be way more usable. Anyhow, I'll look into the specific example...

I was not aware of ReDoc, very cool! (that said, I'm not sure we will be able to get there soon). What do you think then, should we have the ability to flatten complex lists (i.e. lists of dictionaries) ?

[bmartinn]

@S-aiueo32 quick update, it seems like one of the major hurdles is the configuration comparison failing to properly detect a single line change. It will be fixed in the next version. Assuming that, where do you stand on the list in hyper-parameter section issue?

[marekcygan]

@bmartinn I am currently thinking of different ways of configuring experiments, i.e., splitting parts of config file between configuration and hyperparameters. Can you please update me on the issue, in particular:

• is the diff between configurations working better now compared to what @S-aiueo32 pointed out?
• any changes on the way lists of dictionaries are handled in the hyper-parameter section?

[marekcygan]

I checked that diff on configuration works nicely (as of March 21, 2022).