mimir icon indicating copy to clipboard operation
mimir copied to clipboard
verified publisher icon grafana

Flaky TestDispatcherGroupLimits/low_limit

Open bboreham opened this issue 4 years ago • 6 comments

Failed: https://github.com/grafana/mimir/runs/4103684576

--- FAIL: TestDispatcherGroupLimits (4.10s)
    --- FAIL: TestDispatcherGroupLimits/low_limit (3.33s)
        alertmanager_test.go:110: expected <nil>, got 
            metric output does not match expectation; want:
            
            # HELP alertmanager_dispatcher_aggregation_group_limit_reached_total Number of times when dispatcher failed to create new aggregation group due to limit.
            # TYPE alertmanager_dispatcher_aggregation_group_limit_reached_total counter
            alertmanager_dispatcher_aggregation_group_limit_reached_total 4
            
            got:
            
            # HELP alertmanager_dispatcher_aggregation_group_limit_reached_total Number of times when dispatcher failed to create new aggregation group due to limit.
            # TYPE alertmanager_dispatcher_aggregation_group_limit_reached_total counter
            alertmanager_dispatcher_aggregation_group_limit_reached_total 5

Passed on re-run: https://github.com/grafana/mimir/runs/4104035197

bboreham avatar Nov 04 '21 12:11 bboreham

Just encountered this: https://github.com/grafana/mimir/actions/runs/6184952844/job/16789586635?pr=5925

--- FAIL: TestDispatcherGroupLimits (3.28s)
    --- FAIL: TestDispatcherGroupLimits/low_limit (3.03s)
        alertmanager_test.go:130: expected <nil>, got 
            
            Diff:
            --- metric output does not match expectation; want
            +++ got:
            @@ -2,3 +2,3 @@
             # TYPE alertmanager_dispatcher_aggregation_group_limit_reached_total counter
            -alertmanager_dispatcher_aggregation_group_limit_reached_total 4
            +alertmanager_dispatcher_aggregation_group_limit_reached_total 5

Works locally (and I didn't modify anything that would affect this), and passed on a rerun: https://github.com/grafana/mimir/actions/runs/6184952844/job/16790267619?pr=5925

zenador avatar Sep 14 '23 11:09 zenador

Still flaky:

$ go test -run=TestDispatcherGroupLimits -count=1000 ./pkg/alertmanager
--- FAIL: TestDispatcherGroupLimits (3.05s)
    --- FAIL: TestDispatcherGroupLimits/low_limit (3.03s)
        alertmanager_test.go:130: expected <nil>, got 
            
            Diff:
            --- metric output does not match expectation; want
            +++ got:
            @@ -2,3 +2,3 @@
             # TYPE alertmanager_dispatcher_aggregation_group_limit_reached_total counter
            -alertmanager_dispatcher_aggregation_group_limit_reached_total 4
            +alertmanager_dispatcher_aggregation_group_limit_reached_total 5
             
--- FAIL: TestDispatcherGroupLimits (3.01s)
    --- FAIL: TestDispatcherGroupLimits/low_limit (3.01s)
        alertmanager_test.go:130: expected <nil>, got 
            
            Diff:
            --- metric output does not match expectation; want
            +++ got:
            @@ -2,3 +2,3 @@
             # TYPE alertmanager_dispatcher_aggregation_group_limit_reached_total counter
            -alertmanager_dispatcher_aggregation_group_limit_reached_total 4
            +alertmanager_dispatcher_aggregation_group_limit_reached_total 5
             
--- FAIL: TestDispatcherGroupLimits (3.02s)
    --- FAIL: TestDispatcherGroupLimits/low_limit (3.02s)
        alertmanager_test.go:130: expected <nil>, got 
            
            Diff:
            --- metric output does not match expectation; want
            +++ got:
            @@ -2,3 +2,3 @@
             # TYPE alertmanager_dispatcher_aggregation_group_limit_reached_total counter
            -alertmanager_dispatcher_aggregation_group_limit_reached_total 4
            +alertmanager_dispatcher_aggregation_group_limit_reached_total 5
             
--- FAIL: TestDispatcherGroupLimits (3.02s)
    --- FAIL: TestDispatcherGroupLimits/low_limit (3.01s)
        alertmanager_test.go:130: expected <nil>, got 
            
            Diff:
            --- metric output does not match expectation; want
            +++ got:
            @@ -2,3 +2,3 @@
             # TYPE alertmanager_dispatcher_aggregation_group_limit_reached_total counter
            -alertmanager_dispatcher_aggregation_group_limit_reached_total 4
            +alertmanager_dispatcher_aggregation_group_limit_reached_total 5
             
--- FAIL: TestDispatcherGroupLimits (3.03s)
    --- FAIL: TestDispatcherGroupLimits/low_limit (3.02s)
        alertmanager_test.go:130: expected <nil>, got 
            
            Diff:
            --- metric output does not match expectation; want
            +++ got:
            @@ -2,3 +2,3 @@
             # TYPE alertmanager_dispatcher_aggregation_group_limit_reached_total counter
            -alertmanager_dispatcher_aggregation_group_limit_reached_total 4
            +alertmanager_dispatcher_aggregation_group_limit_reached_total 5
             
--- FAIL: TestDispatcherGroupLimits (3.03s)
    --- FAIL: TestDispatcherGroupLimits/low_limit (3.02s)
        alertmanager_test.go:130: expected <nil>, got 
            
            Diff:
            --- metric output does not match expectation; want
            +++ got:
            @@ -2,3 +2,3 @@
             # TYPE alertmanager_dispatcher_aggregation_group_limit_reached_total counter
            -alertmanager_dispatcher_aggregation_group_limit_reached_total 4
            +alertmanager_dispatcher_aggregation_group_limit_reached_total 5
             
--- FAIL: TestDispatcherGroupLimits (3.01s)
    --- FAIL: TestDispatcherGroupLimits/low_limit (3.01s)
        alertmanager_test.go:130: expected <nil>, got 
            
            Diff:
            --- metric output does not match expectation; want
            +++ got:
            @@ -2,3 +2,3 @@
             # TYPE alertmanager_dispatcher_aggregation_group_limit_reached_total counter
            -alertmanager_dispatcher_aggregation_group_limit_reached_total 4
            +alertmanager_dispatcher_aggregation_group_limit_reached_total 5
             
FAIL
FAIL	github.com/grafana/mimir/pkg/alertmanager	28.314s
FAIL

colega avatar Oct 11 '23 11:10 colega

Another example:

--- FAIL: TestDispatcherGroupLimits (3.25s)
    --- FAIL: TestDispatcherGroupLimits/low_limit (3.03s)
        alertmanager_test.go:130: expected <nil>, got 
            
            Diff:
            --- metric output does not match expectation; want
            +++ got:
            @@ -2,3 +2,3 @@
             # TYPE alertmanager_dispatcher_aggregation_group_limit_reached_total counter
            -alertmanager_dispatcher_aggregation_group_limit_reached_total 4
            +alertmanager_dispatcher_aggregation_group_limit_reached_total 5

charleskorn avatar Oct 19 '23 05:10 charleskorn

https://github.com/grafana/mimir/actions/runs/6735289122/job/18308218253?pr=6544

--- FAIL: TestDispatcherGroupLimits (3.29s)
    --- FAIL: TestDispatcherGroupLimits/low_limit (3.03s)
        alertmanager_test.go:130: expected <nil>, got 
            
            Diff:
            --- metric output does not match expectation; want
            +++ got:
            @@ -2,3 +2,3 @@
             # TYPE alertmanager_dispatcher_aggregation_group_limit_reached_total counter
            -alertmanager_dispatcher_aggregation_group_limit_reached_total 4
            +alertmanager_dispatcher_aggregation_group_limit_reached_total 5

zenador avatar Nov 02 '23 16:11 zenador

Still flaky.

colega avatar Aug 05 '24 07:08 colega

Another occurrence here:

--- FAIL: TestDispatcherGroupLimits (3.22s)
    --- FAIL: TestDispatcherGroupLimits/low_limit (3.03s)
        alertmanager_test.go:141: expected <nil>, got 
            
            Diff:
            --- metric output does not match expectation; want
            +++ got:
            @@ -2,3 +2,3 @@
             # TYPE alertmanager_dispatcher_aggregation_group_limit_reached_total counter
            -alertmanager_dispatcher_aggregation_group_limit_reached_total 4
            +alertmanager_dispatcher_aggregation_group_limit_reached_total 5

pracucci avatar Aug 26 '24 08:08 pracucci