skipper icon indicating copy to clipboard operation
skipper copied to clipboard

Published 20 hours ago • verified publisher icon zalando

flaky tests

Open szuecs opened this issue 6 years ago • 8 comments

This could show some error we have in the cluster*Ratelimit filters

7:06:52 PM  time="2019-01-30T18:06:46Z" level=info msg="SWARM: NodeInfo{name: n2-9680, 127.0.0.1:9680} left"
7:06:52 PM  --- FAIL: TestTwoSwarms (4.22s)
7:06:52 PM      --- FAIL: TestTwoSwarms/two_swarms,_few_maxhits (1.30s)
7:06:52 PM          cluster_test.go:383: 2.2 TestTwoSwarmsFewMaxHits backend allowed but should not
7:06:52 PM  FAIL
7:06:52 PM  FAIL	github.com/zalando/skipper/ratelimit	10.531s

szuecs avatar Jan 30 '19 18:01 szuecs

Again

--- FAIL: TestTwoSwarms (4.22s)
    --- FAIL: TestTwoSwarms/two_swarms,_few_maxhits (1.30s)
        swim_test.go:385: 2.2 TestTwoSwarmsFewMaxHits backend allowed but should not

szuecs avatar May 02 '19 09:05 szuecs

Another one:

--- FAIL: TestOAuth2Tokenintrospection (0.53s)
    --- FAIL: TestOAuth2Tokenintrospection/secureOauthTokenintrospectionAllClaim:_valid_token,_one_valid_claim (0.01s)
        tokenintrospection_test.go:598: unexpected status code: 401 != 200
FAIL

--- FAIL: TestOAuth2Tokenintrospection (0.09s)
    --- FAIL: TestOAuth2Tokenintrospection/secureAnyKV():_valid_token,_one_valid_key_value_pair (0.00s)
        tokenintrospection_test.go:598: unexpected status code: 401 != 200
FAIL
FAIL	github.com/zalando/skipper/filters/auth	1.681s

szuecs avatar Jul 05 '19 07:07 szuecs 

--- FAIL: TestOAuth2Tokenintrospection (0.08s)
    --- FAIL: TestOAuth2Tokenintrospection/secureAnyKV():_valid_token,_one_valid_key_value_pair (0.00s)
        tokenintrospection_test.go:598: unexpected status code: 401 != 200
FAIL
FAIL	github.com/zalando/skipper/filters/auth	6.077s

szuecs avatar Jul 08 '19 11:07 szuecs 

--- FAIL: TestWatchInitialRecovers (0.09s)
    watch_test.go:93: failed to load route for: /foo

szuecs avatar Jul 09 '19 19:07 szuecs 

--- FAIL: TestConstantlyUpdatingRoutes (18.18s)
    concurrency_test.go:393: failed to equally balance load, counters: lb group member 6 1341 lb group member 3 1410
    concurrency_test.go:393: failed to equally balance load, counters: lb group member 3 1410 lb group member 6 1341

szuecs avatar Jul 10 '19 08:07 szuecs 

TestConstantlyUpdatingRoutes (13.59s)
535    concurrency_test.go:393: failed to equally balance load, counters: lb group member 2 1350 lb group member 0 1420

szuecs avatar Dec 31 '19 13:12 szuecs

Addressed in: https://github.com/zalando/skipper/pull/1555

~~

TestInitialAndDeleteFile

times out, possible the loggingtest/logger get stuck at https://github.com/zalando/skipper/blob/master/logging/loggingtest/logger.go#L113

Log output:

panic: test timed out after 10m0s

goroutine 60 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.15.3/x64/src/testing/testing.go:1618 +0x11f
created by time.goFunc
	/opt/hostedtoolcache/go/1.15.3/x64/src/time/sleep.go:167 +0x52

goroutine 1 [chan receive, 8 minutes]:
testing.(*T).Run(0xc000001800, 0x13a214e, 0x18, 0x13c7e00, 0x1)
	/opt/hostedtoolcache/go/1.15.3/x64/src/testing/testing.go:1169 +0x5f4
testing.runTests.func1(0xc000001800)
	/opt/hostedtoolcache/go/1.15.3/x64/src/testing/testing.go:1439 +0xa7
testing.tRunner(0xc000001800, 0xc000091ce0)
	/opt/hostedtoolcache/go/1.15.3/x64/src/testing/testing.go:1123 +0x203
testing.runTests(0xc00000f5e0, 0x1b25840, 0x9, 0x9, 0xbfddc294ae0f5f3d, 0x8bb3737c81, 0x1b87e40, 0xc0002547a0)
	/opt/hostedtoolcache/go/1.15.3/x64/src/testing/testing.go:1437 +0x613
testing.(*M).Run(0xc000092580, 0x0)
	/opt/hostedtoolcache/go/1.15.3/x64/src/testing/testing.go:1345 +0x3b4
main.main()
	_testmain.go:61 +0x237

goroutine 54 [select, 8 minutes]:
github.com/zalando/skipper/logging/loggingtest.New.func1(0xc0002b33e0, 0xc0002dc420, 0xc00003b8b8, 0xc0002b3440, 0xc0002b34a0, 0xc0002b3500, 0xc0002b3560, 0xc0002b35c0)
	/home/runner/work/skipper/skipper/logging/loggingtest/logger.go:113 +0x229
created by github.com/zalando/skipper/logging/loggingtest.New
	/home/runner/work/skipper/skipper/logging/loggingtest/logger.go:111 +0x345

goroutine 53 [chan receive, 8 minutes]:
github.com/zalando/skipper/logging/loggingtest.(*Logger).Reset(0xc0002dc450)
	/home/runner/work/skipper/skipper/logging/loggingtest/logger.go:191 +0xa5
github.com/zalando/skipper/eskipfile.(*watchTest).waitAndFailInitial(0xc0002e2700)
	/home/runner/work/skipper/skipper/eskipfile/watch_test.go:156 +0x246
github.com/zalando/skipper/eskipfile.TestInitialAndDeleteFile(0xc000288c00)
	/home/runner/work/skipper/skipper/eskipfile/watch_test.go:247 +0x129
testing.tRunner(0xc000288c00, 0x13c7e00)
	/opt/hostedtoolcache/go/1.15.3/x64/src/testing/testing.go:1123 +0x203
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.15.3/x64/src/testing/testing.go:1168 +0x5bc

goroutine 33 [select]:
github.com/zalando/skipper/routing.receiveFromClient(0x14dc340, 0xc0002dc480, 0xc0002dc4b0, 0x0, 0xe4e1c0, 0xc0002c6700, 0x1, 0x1, 0x0, 0x0, ...)
	/home/runner/work/skipper/skipper/routing/datasource.go:116 +0x399
created by github.com/zalando/skipper/routing.receiveRouteDefs
	/home/runner/work/skipper/skipper/routing/datasource.go:175 +0x258

goroutine 66 [select, 8 minutes]:
github.com/zalando/skipper/routing.receiveRouteDefs.func1(0xc000103d40, 0xc0002b37a0, 0xc000192be0, 0xc0001b7dd0, 0xc000103da0)
	/home/runner/work/skipper/skipper/routing/datasource.go:181 +0x105
created by github.com/zalando/skipper/routing.receiveRouteDefs
	/home/runner/work/skipper/skipper/routing/datasource.go:178 +0x2cd

goroutine 55 [select]:
github.com/zalando/skipper/eskipfile.(*WatchClient).watch(0xc0002dc480)
	/home/runner/work/skipper/skipper/eskipfile/watch.go:132 +0x1c5
created by github.com/zalando/skipper/eskipfile.Watch
	/home/runner/work/skipper/skipper/eskipfile/watch.go:37 +0x1b3

goroutine 56 [select, 8 minutes]:
github.com/zalando/skipper/routing.receiveRouteMatcher(0xc0002dc4b0, 0x0, 0xe4e1c0, 0xc0002c6700, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
	/home/runner/work/skipper/skipper/routing/datasource.go:531 +0x1cc
created by github.com/zalando/skipper/routing.(*Routing).startReceivingUpdates
	/home/runner/work/skipper/skipper/routing/routing.go:334 +0x110

goroutine 57 [select, 8 minutes]:
github.com/zalando/skipper/routing.(*Routing).startReceivingUpdates.func1(0xc0002b3800, 0xc00007d300)
	/home/runner/work/skipper/skipper/routing/routing.go:337 +0x16b
created by github.com/zalando/skipper/routing.(*Routing).startReceivingUpdates
	/home/runner/work/skipper/skipper/routing/routing.go:335 +0x145
FAIL	github.com/zalando/skipper/eskipfile	600.032s
FAIL
/bin/sh: 1: break: Illegal number: -1
make: *** [check-race] Error 2
Makefile:83: recipe for target 'check-race' failed
Error: Process completed with exit code 2.

aryszka avatar Oct 26 '20 10:10 aryszka

Flaky tests per environment, I can locally reproduce only TestFadeIn/random,_7 with about 10% error.

GH actions:

TestConstantlyUpdatingRoutes
TestFadeIn/random,_7

CDP:

TestTwoSwarms/two_swarms,_few_maxhits

szuecs avatar Mar 18 '21 18:03 szuecs

We are now tracking individual flaky tests by label https://github.com/zalando/skipper/issues?q=is%3Aissue+is%3Aopen+label%3Aflakytest

AlexanderYastrebov avatar Oct 10 '23 13:10 AlexanderYastrebov