StressChaos - certificate has expired or is not yet valid

[niroowns]

Bug Report

What version of Kubernetes are you using? 1.28

What version of Chaos Mesh are you using? 2.6.3

What did you do? / Minimal Reproducible Example When applying a StressChaos in a particular cluster, we don't see that the chaos experiment has properly succeeded.

What did you expect to see? Successful completion of the experiment.

What did you see instead? Events: Type Reason Age From Message Normal FinalizerInited 67s initFinalizers Finalizer has been inited Normal Updated 67s initFinalizers Successfully update finalizer of resource Normal Started 67s desiredphase Experiment has started Normal Updated 67s desiredphase Successfully update desiredPhase of resource Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z" Normal Updated 67s records Successfully update records of resource Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z" Normal Updated 67s records Successfully update records of resource Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z" Normal Updated 67s records Successfully update records of resource Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z" Normal Updated 67s records Successfully update records of resource Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z" Normal Updated 67s records Successfully update records of resource Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z" Normal Updated 67s records Successfully update records of resource Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z" Normal Updated 67s records Successfully update records of resource Warning Failed 67s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:58Z is after 2024-01-04T08:54:33Z" Normal Updated 67s records Successfully update records of resource Warning Failed 66s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:59Z is after 2024-01-04T08:54:33Z" Normal Updated 66s records Successfully update records of resource Warning Failed 66s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:47:59Z is after 2024-01-04T08:54:33Z" Normal Updated 66s records Successfully update records of resource Warning Failed 65s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:48:00Z is after 2024-01-04T08:54:33Z" Normal Updated 65s records Successfully update records of resource Warning Failed 65s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:48:00Z is after 2024-01-04T08:54:33Z" Normal Updated 65s records Successfully update records of resource Warning Failed 59s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:48:06Z is after 2024-01-04T08:54:33Z" Normal Updated 59s records Successfully update records of resource Warning Failed 59s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:48:06Z is after 2024-01-04T08:54:33Z" Normal Updated 59s records Successfully update records of resource Warning Failed 39s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:48:26Z is after 2024-01-04T08:54:33Z" Normal Updated 39s records Successfully update records of resource Warning Failed 39s records Failed to apply chaos: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T13:48:26Z is after 2024-01-04T08:54:33Z" Normal Updated 39s records Successfully update records of resource

chaosctl debug output is the same: ./chaosctl debug stresschaos chaosmesh-cpu-stress -N cluster-addons -n cluster-addons failed to execute cmd:

failed get pid from pod xxxxxxx/xxxxx-xxxxxx-7c6b8fd5c-rj2zm: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-05-22T14:49:21Z is after 2024-01-04T08:54:33Z"

I am really perplexed at the root of where the "after" (2024-01-04T08:54:33Z) timestamp is coming from. We use cert-manager and I have validated that all the certificates being used by the controller and the agent (including the corresponding Issuers) are all fine.

Any help is greatly appreciated.