craft icon indicating copy to clipboard operation
craft copied to clipboard

Improve GitHub asset upload retry algorithm

Open chadwhitacre opened this issue 3 years ago • 3 comments

Seen a few times now that we can get an intermittent asset upload failure from GitHub. Do we have a retry already for that? How about it?

@rhcarvalho @iker-barriocanal Do you recall the retry conversation we had earlier? Was it around GitHub asset uploads?

chadwhitacre avatar Dec 15 '21 23:12 chadwhitacre

We try up to 3 times: https://github.com/getsentry/craft/blob/03fda08404939fb2a88c5059e1e5634689ea1cf1/src/targets/github.ts#L389

We can probably increase it and also add some delay?

BYK avatar Dec 16 '21 07:12 BYK

@chadwhitacre it was about asset downloads.

iker-barriocanal avatar Dec 16 '21 12:12 iker-barriocanal

Looking at the code at https://github.com/getsentry/craft/blob/03fda08404939fb2a88c5059e1e5634689ea1cf1/src/targets/github.ts#L394-L396, I am not sure if the retry logic is working as expected. @chadwhitacre's https://github.com/getsentry/publish/runs/4540082749?check_suite_focus=true#step:8:169 did have 3x 500 requests to the failed asset, but I don't see a "Reached maximum retries..." error.

image

We need https://github.com/getsentry/publish/issues/665 to get better stack traces :)

+1 to potentially a few more retries and exponential backoff.

rhcarvalho avatar Dec 16 '21 12:12 rhcarvalho