Add retry to tests

[CharlieTLe]

What this PR does: Adds retry to tests. This should improve the reliability of the tests.

Which issue(s) this PR fixes: Fixes #

Checklist

• [ ] Tests updated
• [ ] Documentation added
• [ ] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[CharlieTLe]

Example retry: https://github.com/cortexproject/cortex/actions/runs/9196226564/job/25293837536?pr=5968

  for run in $(seq 1 3); do
    make BUILD_IN_CONTAINER=false test && break
    echo "Retrying tests... Run $run failed."
  done
  shell: sh -e {0}
go test -tags netgo -timeout 30m -race -failfast -count 1 ./...
?   	github.com/cortexproject/cortex/cmd/test-exporter	[no test files]
...
--- FAIL: TestCompactor_ShouldCompactOnlyShardsOwnedByTheInstanceOnShardingEnabledWithShuffleShardingAndMultipleInstancesRunning (19.70s)
    compactor_test.go:1442: 
        	Error Trace:	/__w/cortex/cortex/pkg/compactor/compactor_test.go:1442
        	Error:      	Received unexpected error:
        	            	group with group_hash=106465857 owned by multiple compactors
        	Test:       	TestCompactor_ShouldCompactOnlyShardsOwnedByTheInstanceOnShardingEnabledWithShuffleShardingAndMultipleInstancesRunning
FAIL
FAIL	github.com/cortexproject/cortex/pkg/compactor	37.803s
ok  	github.com/cortexproject/cortex/pkg/configs/api	1.130s
...
--- FAIL: TestSlottedTicker (2.08s)
    --- FAIL: TestSlottedTicker/Get_last_slot (2.08s)
        time_test.go:234: expected true, got false
FAIL
FAIL	github.com/cortexproject/cortex/pkg/util	26.946s
...
ok  	github.com/cortexproject/cortex/tools/querytee	1.085s
ok  	github.com/cortexproject/cortex/tools/thanosconvert	1.161s
FAIL
make: *** [Makefile:226: test] Error 1
Retrying tests... Run 1 failed.
go test -tags netgo -timeout 30m -race -failfast -count 1 ./...
?   	github.com/cortexproject/cortex/cmd/test-exporter	[no test files]
...
ok  	github.com/cortexproject/cortex/tools/thanosconvert	1.183s

[alanprot]

Im ok with this i guess

[alanprot]

I tried to fix some flaky tests today.

[yeya24]

Actually, I am a little bit concerned. If the test doesn't pass because the code has bug, it will still retry 3 times? I don't know if it is what we want and we will spend more time on CI.

[alanprot]

Yeah.. lets hold this for now and try to make the tests more stable.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.