runner-container-hooks icon indicating copy to clipboard operation
runner-container-hooks copied to clipboard

Implement exponential retry/backoff mechanism in k8s pod exec

Open nielstenboom opened this issue 1 year ago • 1 comments

Hey there!

We've internally been experiencing issues with the following two issues:

https://github.com/actions/runner-container-hooks/issues/103 https://github.com/actions/runner-container-hooks/issues/107

In this PR I attempted to tackle them both by;

  • Making sure the await from exec.exec is wrapped in a try/catch so UnhandledPromiseRejection errors should no longer occur to this
  • I've wrapped the call to k8s.Exec.exec() in an exponential backoff using this package. This should mitigate the rare HttpErrors sometimes being thrown from this function.

Typescript is not my main language so apologies if things are not entirely how they should be, did my best to add some proper tests for it 💪

Please please don't hesitate to make (major) edits to this PR! I'd be very happy if that would close these issues quicker 😄

Thanks!

closes https://github.com/actions/runner-container-hooks/issues/103 closes https://github.com/actions/runner-container-hooks/issues/107

nielstenboom avatar Dec 21 '23 13:12 nielstenboom

@nikola-jokic would you have time to look at this soon? Thanks 😄

nielstenboom avatar Jan 08 '24 13:01 nielstenboom