"The usage quota was exceeded: type=metrics_storage"

[Wonder1905]

Hi, after ClearML working great for two weeks I'm getting the following error: clearml.log - WARNING - failed logging task to backend (2 lines, <400/68: events.add_batch/v1.0 (The usage quota was exceeded: type=metrics_storage)>) And: 2022-03-27 10:48:02,642 - clearml.Metrics - ERROR - Action failed <400/68: events.add_batch/v1.0 (The usage quota was exceeded: type=metrics_storage)> I deleted tons of experiments, but without any change. Are the metrics just the plotted scalars? or also the checkpoints being saved? I'll be happy for your help. Thanks.

[jkhenning]

Hi @BattashB,

The metric events are basically reported scalars, plots and logs - when did you delete these experiments? The usage quota is currently updated once per day

[igor-i-m]

Hello. I've encountered the same problem as BattashB. My data usage quota for metrics exceeded(1.26 GB/1GB) and i am getting the same error as mentioned above: 2022-08-29 10:23:37,234 - clearml.log - WARNING - failed logging task to backend (1 lines, <400/68: events.add_batch/v1.0 (The usage quota was exceeded: type=metrics_storage)>) (don't mind date, checked that problem is still there) After I started getting this error I archived and deleted all Tasks, but it's been 2 days without any change in data usage for metrics, quota is still reached. Any tips? Thanks in advance.

[oren-allegro]

Hi @igor-i-m,

I will try to take a look to see why the metrics storage wasn't clear. Could you share your workspace id (can be found in the top right part of the settings > workspace page)? Alternatively, you can DM me in the clearml slack channel.

[igor-i-m]

@oren-allegro id is e8904b5404ca4a6285bdbc0d4bf8b623

[oren-allegro]

@igor-i-m - I found the cause of the issue you are facing. The data was deleted, but there is a problem in the storage calculation in the hosted SaaS (not in the clearml project). I temporarily added more storage to your workspace until this issue is resolved. If you have any additional issues related to the hosted service - I would advise you to use the slack channel - where there is a channel for discussing SaaS related issues. You can join the clearml slack using this link.

[Akshay1-6180]

I am facing this same issue , I deleted a lot of the tasks but it still shows the storage is exceeded , @oren-allegro could you check and let me know what the issue is , my id is : fc492a8668af46b598c78500238d9d04.

[Robotatron]

is there a way to delete metadata from the UI? Right now I only see how to archive the runs, but not delete

[jkhenning]

@Robotatron if you go into the archive, you can delete the runs there