clearml icon indicating copy to clipboard operation
clearml copied to clipboard

Cannot use OutputModel.report_confusion_matrix before OutputModel.update_weights

Open HarryGoodman opened this issue 1 year ago • 1 comments

Describe the bug

I am trying to report a confusion matrix for my Output Model but it seems that if I try and do this before running OutputModel.update_weights it gives a warning and does not throw an error.

To reproduce

`task = Task.init(project_name=params['project name'], task_name= "-".join([params['task name'], params['net']['model'], 'lr', str(params['opt']['lr'])]), auto_connect_frameworks={"tensorflow":False,"tensorboard": False,'pytorch': False,})

output_model = OutputModel(task=task, framework="PyTorch", config_dict = params, tags = [params['project name']] + params['tags'],)

[...]

output_model.report_confusion_matrix(title = "Validation Confusion Matrix", series = "Validation Data", matrix = best_total_cm, xaxis = "Predicted Labels", yaxis = "True Labels", xlabels = list(classes.keys()), ylabels = list(classes.keys()),)

output_model.update_weights( weights_filename=os.path.join(params["output dir"],'model_best.pth'), auto_delete_file=False, ) `

Expected behaviour

I get the following warning: 2023-07-18 14:15:34,407 - clearml.Metrics - ERROR - Action failed <400/131: events.add_batch/v1.0 (Events not added: Event must have a 'task' field=1)> The confusion matrix does not upload to the clearml server. However, when I swap the report_confusion_matrix and update_weights I do not get the warning and the confusion matrix is uploaded.

Environment

  • Server type: self hosted
  • ClearML: 1.11.0
  • ClearML Server Version; WebApp: 1.10.0-357 • Server: 1.10.0-357 • API: 2.24
  • Python Version: 3.8.16
  • OS: Ubuntu 20.04.5 LTS (Focal Fossa)

Related Discussion

https://clearml.slack.com/archives/CTK20V944/p1689654081105309

HarryGoodman avatar Jul 19 '23 01:07 HarryGoodman

Hey @HarryGoodman, we just released ClearML SDK 1.12.2 which should fix this issue. Can you please try it and tell us whether it works now?

AlexandruBurlacu avatar Aug 11 '23 22:08 AlexandruBurlacu