cortex-tools
cortex-tools copied to clipboard
Analyze cortex failure
To avoid large queries, we've set a limit to the number of series a query can hit
./cortextool analyse prometheus \
--address=https://internal-cortex.../ \
--id=<redacted> \
--key=<redacted> \
--log.level=debug \
--grafana-metrics-file="metrics-in-grafana.json"
DEBU[0025] in use cortex_request_duration_seconds_count 2
DEBU[0025] in use cortex_ring_members 60
DEBU[0025] in use cortex_s3_request_duration_seconds_count 0
DEBU[0025] in use cs_error_counter 3
DEBU[0025] in use db_mat_view_query_duration_count 750
cortextool: error: error querying count by (job) (db_model_duration_bucket): execution: expanding series: the query hit the max number of series limit (limit: 100000 series), try --help
is there anyway we can just skip this metrics rather than have the whole analysis fail?