Provide ability to wipe off data for AMP workspace/metrics

[datsabk]

Use case 1

A Prometheus Counter metrics is an incremental metric that takes a new datapoint and adds/subtracts value to existing counter value. In scenarios where we wish to reset this counter, presently the only solution to use a new metrics altogether which calls for changes across dashboards and alerts.

Use case 2

In scenarios where due to an internal implementation error we end up pushing metrics which PII data in the tags. This data, when detected, needs to be wiped off immediately by wiping off tsdb data by duration/metric. In such scenarios, destroying workspace and losing on all metric data is not an option.

Use case 3

When we reach Quota limits - we presently get 400 Bad request errors for two hours as mentioned on AMP quota limits page

metrics that have reported data in the past 2 hours

Feature Request

Provide the ability to wipe off Prometheus Data on demand by metric or just a complete wipeoff by age of data. This will allow us to quickly unblock metric submission instead of losing on 2 hours of data altogether. Destroying and recreating workspace leads to a lot of changes which are not practical.