Repeated failures uploading binaries, `curl: (55) OpenSSL SSL_write: Connection reset by peer, errno 104`

[rfay]

I have trouble every time I do a release because it will work through some of the binaries, then fail on one with

curl: (55) OpenSSL SSL_write: Connection reset by peer, errno 104

I assume this is a GitHub infrastructure problem, and I've actually raised an issue before, but they didn't answer for weeks and then said nothing useful.

What I have to do every time is delete the existing artifacts that have successfully been attached to the release, then save, then restart the job. It's very painful. Right now on https://github.com/drud/ddev/releases/tag/v1.19.1 I'm on my 9th try, see https://github.com/drud/ddev/actions/runs/2016085486

Is there something that I could do better than https://github.com/drud/ddev/blob/43f59cbaadc15be8cc70baa0293beff954d159a2/.github/workflows/master-build.yml#L234-L240 ? Maybe upload artifacts one at a time and sleep between them or something? Could github-action-public-binaries try twice? (It would mean having to go delete the partially created one that failed).

Thanks for your help. I imagine you've been around this more than I have, I just have to battle it on every release.

[skx]

(Sorry for the delay in getting back to you, a busy month for me!)

I have hit sporadic failures in the past, and generally I do the same thing as you:

• Delete any uploads which are zero-bytes, or obviously incomplete.
• Click "rerun" / "retry" on the action.

It is annoying, and I think allowing the uploads to retry a couple of times before admitting defeat is a good idea. The only potential caveat I see is that if an upload is interrupted mid-way there might need to be an explicit "remove artifact" step which I'm not sure immediately how to handle.

I'll have a think on it, and see what the retry behaviour looks like on failure..

[rfay]

Thanks for the response. Yes, it's pretty painful.

[varshavaradarajan]

@skx - can we not do curl --retry?

Edit: curl retry with -C -?

Use "-C -" to tell curl to automatically find out where/how to resume the transfer. It then uses the given output/input files to figure that out.

[skx]

I've been sitting on this issue for a while now, which I appreciate must be a bit frustrating for people who get hit by problems.

Unfortunately @varshavaradarajan adding retry will not be sufficient based on the testing that I've done (outside this repository).

What tends to happen on failure is:

• curl runs and starts the artifact upload.
• It uploads a bit of a binary, then the connection dies.

The end result is a zero-byte, or otherwise incomplete, upload.

Retrying the upload a second time then fails with the predictable "This artifact is already present" error.

So I think to solve this issue the actual solution must be:

• For each artifact:
  • Delete if already present.
  • Upload (again).

Skipping artifacts which are already present, by silently failing, could also be a bit of a horrid solution. The end result of that might be that there are ten artifacts, but one/two of them are incomplete/broken. Not ideal, but perhaps progress.

That said I've seen fewer failures recently. e.g. I just made a new release of my aws-utils project, and there was no need to delete artifacts and re-run the failed action. The ideal case is of course that we cope with errors, but if the github API is getting more reliable that's a nice bonus!

[bbimber]

Our workflow just started to intermittently hit this as well. Our workflow in question uploads two artifacts, one ~250mb and one under 100. Some of the time they both work, sometimes the first fails, and sometimes the first works and the second fails. It's not really clear what is causing the failures, but if there was a either more fault-tolerant way to upload, or perhaps greater debugging information that could be output that'd be helpful.

Echoing the other thread, some kind of 'delete-if-exists' functionality would be helpful both in general (like a nightly build that always overwrites the previous artifacts), or resuming a job like this when it uploads multiple artifacts.

[ThFriedrich]

This problem is bugging me for a while now too. Any update on this issue yet? Is anyone actively working on this at this point or has anyone found a viable workaround?

[skx]

I don't believe anybody external is working on this, and while it annoys me at times I've not found the time to make any changes myself.

To recap:

• Retrying failed uploads is trivial. But won't be sufficient.
• Incomplete uploads must first be identified, and those assets removed, otherwise a second upload will fail "Already present".

95% of the time the action works for me, as-is. The few times it fails I delete the assets (manually) and rerun the action (manually).

Not ideal, but I can live with it. Might be worth searching the "marketplace" and seeing if there's a better alternative which already implements the upload/remove-on-failure/etc support. I've not looked recently, though at the time I put this hack together I think it was the only such action present.