How to access hyper parameters that are in configuration objects for hyperparameter optimization

[ernestlwt]

I am currently using MMDetection and tracking the results on Clearml. The hyper parameters are not located under <Hyperparameter>/<args or general> but in <configuration objects>/<config> instead.

Following the guide, i am stuck at this portion where i am having difficulty accessing them

https://github.com/allegroai/clearml/blob/b9b0a506f35a414f6a9c2da7748f3ec3445b7d2d/examples/optimization/hyper-parameter-optimization/hyper_parameter_optimizer.py#L62-L73

Any solution/workaround to proceed on this? Thanks

[DavidNativ]

Hi you could retrieve what is logged in the configuration section using task.get_configuration_object_as_dict(name). The output should be an OrderedDict.

[ernestlwt]

Hi, i might be missing something here, but how do i reference the hyperparameters that are logged in the configuration section with the class UniformIntegerParameterRange since it takes in a string?

I have tried using task.get_configuration_object_as_dict(name) but it was unable to parse the configs correctly and results in ValueError("Could not parsee configuration text"). I could use task.get_configuration_objects() instead and process the output manually but that will not solve the problem of not being able to reference it with the classes as shown in the example(UniformIntegerParameterRange and DiscreteParameterRange) to do hyperparameter optimization

[erezalg]

Hi @ernestlwt, Calling this works for me: task.get_configuration_object_as_dict(name='General')['parameter_optimization_space']

I get a list of configuration objects which I can parse, can you give it a try?

[ernestlwt]

Hi @erezalg,

Unfortunately that did not work for me. The return of task.get_configuration_object_as_dict(name='General') is none. Were you using mmdetection with clearml too?

[erezalg]

hi @ernestlwt,

No, I was trying with the general HPO code. Could you maybe share a short code snippet with the task declaration and the parameter input? I'll try to help you with that.

[ernestlwt]

Thanks in advance,

using the screenshot as an example,

My hyperparameter lies in Configuration Objects > Config(which is General in this screenshot), rather than Hyper Parameters > General/Args when i am training using MMDetection with clearml.

Code snippet of what i am doing now:

cl_task = Task.init(
  project_name=args.proj_name,
  task_name=args.optimizer_task_name,
  task_type=Task.TaskTypes.optimizer,
  output_uri=args.clml_output_uri
) # no issues

task = Task.get_task(project_name=args.proj_name, task_name=args.training_task_name) # no issues

an_optimizer = HyperParameterOptimizer( # error here
    base_task_id=task.id,
    # example of error:
    # unable to reference the batch_size hyperparameter here because my batch_size parameters location is not @
    # <Hyper Parameters>/<General>/batch_size so i cant referenced it as 'General/batch_size'
    # but is in the config file and is read on clearml as
    # <Configuration Objects>/config/<a part of the dictionary>
    hyper_parameters=[
        UniformIntegerParameterRange('General/layer_1', min_value=128, max_value=512, step_size=128),
        UniformIntegerParameterRange('General/layer_2', min_value=128, max_value=512, step_size=128),
        DiscreteParameterRange('General/batch_size', values=[96, 128, 160]),
        DiscreteParameterRange('General/epochs', values=[30]),
    ],
    objective_metric_title='epoch_accuracy',
    objective_metric_series='epoch_accuracy',
    objective_metric_sign='max',
    max_number_of_concurrent_tasks=2,
    optimizer_class=aSearchStrategy,
    execution_queue=execution_queue,
    spawn_project=None,  # 'HPO spawn project',
    save_top_k_tasks_only=None,  # 5,
    time_limit_per_job=10.,
    pool_period_min=0.2,
    total_max_jobs=10,
    min_iteration_per_job=10,
    max_iteration_per_job=30,
)

[erezalg]

Hi @ernestlwt,

I think I now understand what you mean :) you want the HPO process to change parameters inside a configuration object and not hyperparameters, am I correct? I am actually not 100% sure this is possible, but I'll have to double check.

Is there a reason why your hyperparameters are in a configuration object and not in a hyperparameter section? Are they read from a file?

[ernestlwt]

Yes that is correct.

MMDetection's configurations are all in .py files rather than the argparse method, which i assumed is the reason they are in the configuration object section.

[erezalg]

Hi @ernestlwt,

Sorry for the slow response!

Unfortunately, you can't do HPO on configuration objects. We can add that (as long as the object contains a json \ YAML for example) but this needs to be added.

Just so I understand, can you attach a script that reads a configuration object as such so I can see that it is indeed feasible?

Thanks!

[ernestlwt]

No worries. Thanks for checking it out.

I do not get what you mean. Are you suggesting that the .py configuration file reads a .json or .yaml for the hyper parameters that will be required for optimization?

Alternatively, I am also thinking that I could do a workaround and argparse the hyper parameter values and set it to environment variables and have my .py configuration file read those environment variables directly. I have not tried this out though. What do you think about this approach?

[erezalg]

What I understand is that the parameters in MMDetection are saved in a configuration object and not in a hyperparameter. What I'm saying is that configuration object doesn't impose a strict structure, you can put whatever text you want there. That being said, if the structure is json or yaml inside, we can parse it and in the future add support for doing parameter search on it.

I think your approach is better, it would support the built-in functionality of clearml and you could use HPO with this method.