pulumi-eks icon indicating copy to clipboard operation
pulumi-eks copied to clipboard

Fargate tests are flakey

Open rquitales opened this issue 1 year ago • 1 comments

What happened?

Surfaced in #1039. Our daily cron job fails about 33% of the time due to our Fargate tests.

The error from the test is:

 utils.go:290: 
          	Error Trace:	/home/runner/work/pulumi-eks/pulumi-eks/examples/utils/utils.go:290
          	            				/home/runner/work/pulumi-eks/pulumi-eks/examples/utils/utils.go:225
          	            				/home/runner/work/pulumi-eks/pulumi-eks/examples/utils/utils.go:84
          	            				/home/runner/work/pulumi-eks/pulumi-eks/examples/utils/utils.go:90
          	            				/opt/hostedtoolcache/go/1.21.6/x64/src/runtime/asm_amd64.s:1650
          	Error:      	Not equal: 
          	            	expected: 3
          	            	actual  : 2
          	Test:       	TestAccFargate
          	Messages:   	2 out of 3 Pods are ready

It is unclear why only 2 of the 3 pods can become live and requires further debugging.

Example

https://github.com/pulumi/pulumi-eks/issues/1039

Output of pulumi about

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

rquitales avatar Feb 22 '24 00:02 rquitales

@rquitales I think we also need to disable Fargate tests until this is resolved, otherwise we'll constantly get new P1s like this #1043.

thomas11 avatar Feb 22 '24 07:02 thomas11