cortex-tools icon indicating copy to clipboard operation
cortex-tools copied to clipboard

Analyze cortex failure

Open flipsed opened this issue 1 year ago • 0 comments

To avoid large queries, we've set a limit to the number of series a query can hit

./cortextool analyse prometheus \
    --address=https://internal-cortex.../ \
    --id=<redacted> \
    --key=<redacted> \
    --log.level=debug \
    --grafana-metrics-file="metrics-in-grafana.json"
DEBU[0025] in use cortex_request_duration_seconds_count 2 
DEBU[0025] in use cortex_ring_members 60                
DEBU[0025] in use cortex_s3_request_duration_seconds_count 0 
DEBU[0025] in use cs_error_counter 3                    
DEBU[0025] in use db_mat_view_query_duration_count 750  
cortextool: error: error querying count by (job) (db_model_duration_bucket): execution: expanding series: the query hit the max number of series limit (limit: 100000 series), try --help

is there anyway we can just skip this metrics rather than have the whole analysis fail?

flipsed avatar Sep 28 '23 05:09 flipsed