Trial failed with value None - ClearML Optuna integration crashes with TypeError: 'NoneType' object is not subscriptable

When running a ClearML automation pipeline using Optuna for hyperparameter optimization, the process crashes mid-study with a TypeError. It appears related to the return value of the objective function, possibly due to None being returned and accessed as a subscriptable type.

WebApp: 2.0.0-613 • Server: 2.0.0-613 • API: 2.31 OS: [Ubuntu 20.04 / Docker container]

[W 2025-04-11 15:42:30,086] Trial 36 failed with parameters: {'params_mmahist/model_params/T': 119, 'params_mmahist/model_params/phi': 1075223.4721101117, 'params_mmahist/model_params/mu': 0.95367591135294, 'params_mmahist/model_params/gamma': 2.1638089728988703, 'params_mmahist/risk_params/ticks_per_level': 109, 'params_mmahist/risk_params/bucket_size': 222, 'params_mmahist/risk_params/n_grid': 6} because of the following error: TypeError("'NoneType' object is not subscriptable").
Traceback (most recent call last):
  File "/root/.clearml/venvs-builds/3.12/task_repository/quantmakers.git/.venv/lib/python3.12/site-packages/optuna/study/_optimize.py", line 197, in _run_trial
    value_or_values = func(trial)
                      ^^^^^^^^^^^
  File "/root/.clearml/venvs-builds/3.12/task_repository/quantmakers.git/.venv/lib/python3.12/site-packages/clearml/automation/optuna/optuna.py", line 92, in objective
    objective_metric = objective_metric[0]
                       ~~~~~~~~~~~~~~~~^^^
TypeError: 'NoneType' object is not subscriptable
[W 2025-04-11 15:42:30,086] Trial 36 failed with value None.

OptunaObjective result metric=-0.012996431021644633, iteration (0, -0.012996431440114975)
[I 2025-04-11 19:04:11,847] Trial 26 finished with value: -0.012996431021644633 and parameters: {'params_mmahist/model_params/T': 557, 'params_mmahist/model_params/phi': 13721596.735374013, 'params_mmahist/model_params/mu': 0.6647527064824519, 'params_mmahist/model_params/gamma': 2.089428557356418, 'params_mmahist/risk_params/ticks_per_level': 453, 'params_mmahist/risk_params/bucket_size': 33, 'params_mmahist/risk_params/n_grid': 7}. Best is trial 17 with value: 0.004167059521320642.
Exception in thread Thread-1 (_daemon):
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/root/.clearml/venvs-builds/3.12/task_repository/quantmakers.git/.venv/lib/python3.12/site-packages/clearml/automation/optimization.py", line 1923, in _daemon
    self.optimizer.start()
  File "/root/.clearml/venvs-builds/3.12/task_repository/quantmakers.git/.venv/lib/python3.12/site-packages/clearml/automation/optuna/optuna.py", line 198, in start
    self._study.optimize(
  File "/root/.clearml/venvs-builds/3.12/task_repository/quantmakers.git/.venv/lib/python3.12/site-packages/optuna/study/study.py", line 475, in optimize
    _optimize(
  File "/root/.clearml/venvs-builds/3.12/task_repository/quantmakers.git/.venv/lib/python3.12/site-packages/optuna/study/_optimize.py", line 100, in _optimize
    f.result()
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.clearml/venvs-builds/3.12/task_repository/quantmakers.git/.venv/lib/python3.12/site-packages/optuna/study/_optimize.py", line 160, in _optimize_sequential
    frozen_trial = _run_trial(study, func, catch)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.clearml/venvs-builds/3.12/task_repository/quantmakers.git/.venv/lib/python3.12/site-packages/optuna/study/_optimize.py", line 248, in _run_trial
    raise func_err
  File "/root/.clearml/venvs-builds/3.12/task_repository/quantmakers.git/.venv/lib/python3.12/site-packages/optuna/study/_optimize.py", line 197, in _run_trial
    value_or_values = func(trial)
                      ^^^^^^^^^^^
  File "/root/.clearml/venvs-builds/3.12/task_repository/quantmakers.git/.venv/lib/python3.12/site-packages/clearml/automation/optuna/optuna.py", line 92, in objective
    objective_metric = objective_metric[0]
                       ~~~~~~~~~~~~~~~~^^^
TypeError: 'NoneType' object is not subscriptable
2025-04-11 19:04:12,656 [INFO] optimize.optimize - Top Experiments: ['d4a8eb45ca0b410995c1313385f664e5', 'a18c3879f052429baa94bc7ea12d614d', '3c2b4b8993234552977c232ed9416238']
`Study.stop` is supposed to be invoked inside an objective function or a callback.

@isslerman Which clearml package version is in use?

Hi @isslerman ! I believe you are getting this error because ClearML could not find the objective under the task's reported scalars. The error should indeed be informative. Make sure the metric you want to base your optimization on is actually present at the end of each task to avoid this behaviour.