hub-feedback
hub-feedback copied to clipboard
docker
Docker auto build stuck forever
Problem description
repository: https://hub.docker.com/r/darrenhoo/centos5-go-bootstrap
Build log shows that it is finished, but its status is in in progress
Finnally, it finished:
Created: January 7, 2020 02:37 AM Finished: January 7, 2020 10:37 AM
Queue time: 240m Build time: 240m
Bumping to reopen this. I'm experiencing this today, and
Queue time: 240m
Build time: 240m
looks pretty fishy. Seems like a build completed hook somewhere just isn't working.
Seeing my build stuck today as well, 4 hours now. Also noticing duplicate builds when I push to github
Same have this currently, build complete but still stuck holding up other pushes.
Mine just finally completed like 4 hours after.
It was probably my fault tbh I'd cancelled the build half way through building.
Same have this currently, build complete but still stuck holding up other pushes.
Not sure if it is related to my issue, but somehow I had a second hook created in github and each build was triggering two times. I'm not 100% sure that was related to the build being hung up or not, but when I removed that it seemed to get better.
Currently seeing this in one of my builds, going on for 2 hours but it usually finishes in 15 minutes.
Edit: it finished now
Same have this currently, build complete but still stuck holding up other pushes.
Not sure if it is related to my issue, but somehow I had a second hook created in github and each build was triggering two times. I'm not 100% sure that was related to the build being hung up or not, but when I removed that it seemed to get better.
I have removed duplicate webhooks some time ago, but i'mt still experience this issue. I had this issue on the 23rd April and again today (3rd June). According to the build logs the build finished, but it get's stuck in progress. Cancelling does not help. I sent an email to support on both occasions.
Mine was stuck because I'd cancelled it at an in-opportune time in the build process I gather.
I didn't have any second webhooks created anyway.
Piling on. Build finished in 1 minute, build showed in progress for several hours after the build completed, so I hit cancel. Several hours later it's still trying to cancel the build. In short, a 1 minute build has been "running" for 4 hours now.
This issue comes back every few months... the rate of regression on this issue is horrible.
I imagine this isnt meant to be a proper deployment solution, I have been using it as such for a hobby project where Watchtower just updates my containers when a new version of the image is available, but perhaps we should just see it as "eventually available"
Just got bit by a variant of this on https://hub.docker.com/r/relaysh/pulumi-step-run The build against 9c03e9e finished and pushed successfully according to the log, where the last message was "Build finished" but it was still in progress according to the dashboard. I clicked "Cancel" and now it's in "Cancelling" status and has been for several hours. Further builds are stacked up in "Pending" waiting for this cancellation.
This is clearly an issue when you cancel a build that has already failed or completed. Not sure why this can't be targeted.
I have this issue, a C# dotnetcore build failed "DockerBuild -o /app/build' returned a non-zero code: 1" after that it build hanged.
workaround: Delete the docker hub repo and recreate with a new name side-effects: Anyone using the old repo name will be forced to update
same issue for me, happened twice over 2 weeks. The last time I turned down my remote instance which pulls new images using watchdog, and I noticed it didn't take as long. I wonder if something to do with remote images trying to pull in while its being built? I did notice when I manually tried pulling in new image during time that docker repo was stalled, my remote machine did download the new image (took about an hour though for a 2 min. job normally).
Hi everyone, thanks for your feedback.
Along with the recent comments here, we have had a few support tickets come in recently regarding hanging builds. The builds may time out after four hours after becoming stuck. We fixed one cause in September, but I see there have been quite a few reports from everyone after that point.
I've created a sprint ticket on our end to investigate.
@shawnaxsom This is still very much an issue - do you have any updates?
Sorry all, @dylankbuckley and @alexks02, work on this issue has been pushed to the current sprint. It's earmarked for one of our engineers to start on after he finishes up his current tasks this sprint, hopefully soon. I'm not sure how difficult of an issue it will be to reproduce and address, so I can't guarantee a timeline, but it is a priority for us.
It may have been a little worse today as processing events backed up slightly during one of our deployments. That engineer is helping get the processing sped up for today's processing build events. I'll share your comments here with him.
How often has this been happening lately?
We don't think there is a bug that is causing the issue. The comments we've seen in this thread have primarily been a queuing issue, where the queue of builds becomes backed up.
We've changed the services infrastructure in the past month, and we've been addressing some bottlenecks in our overall infrastructure that hopefully have lowered the number of times this has happened to you recently (besides the queue backup 4 days ago). Let me know otherwise if it is happening often (and at what frequency would help).
Can’t say I’ve had it recently because I avoid it like the plague, but for me it was easy to replicate: accidentally push a branch when it was already building. Triggering another build.