docker-alpine
docker-alpine copied to clipboard
gliderlabs
Temporary error (try again later)
I run into this issue when building a docker image on Ubuntu Host. The Same build on Windows 10 using docker-tools CLI (so technically inside a VirtualBox VM) has no issues and Docker images are built correctly.
I have reinstalled Docker on my Ubuntu completely to have a fresh version just in case. Any suggestions what else I can do to troubleshoot this issue?
Build command output
Sending build context to Docker daemon 572.4kB
Step 1/15 : FROM alpine:edge
edge: Pulling from library/alpine
cc5efb633992: Pull complete
Digest: sha256:2b796ae57cb164a11ce4dcc9e62a9ad10b64b38c4cc9748e456b5c11a19dc0f3
Status: Downloaded newer image for alpine:edge
---> f96c4363411f
Step 2/15 : RUN apk add --update nodejs
---> Running in 12865b082f34
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/edge/main: temporary error (try again later)
WARNING: Ignoring APKINDEX.066df28d.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/edge/community: temporary error (try again later)
WARNING: Ignoring APKINDEX.b53994b4.tar.gz: No such file or directory
ERROR: unsatisfiable constraints:
nodejs (missing):
required by: world[nodejs]
The command '/bin/sh -c apk add --update nodejs' returned a non-zero code: 1
My dockerfile
FROM alpine:edge
RUN apk add --update nodejs
#create an app directory
RUN mkdir -p /app
WORKDIR /app
# create some useful folders
RUN mkdir -p /data
# Copy apps build in another folder
COPY main.js /app
COPY package.json /app
COPY version.json /app
COPY dist /app/dist
COPY data /data
COPY app_modules /app/app_modules
COPY node_modules /app/node_modules
#expose the port
EXPOSE 5555
#start the app
#ENV NODE_ENV=development
#ENV debug=true
WORKDIR /app
# comment if need a console in the container
CMD [ "node", "main.js" ]
This keeps popping up in other threads and I wonder if there might be a change in the way Fastly is doing DNS. To debug this we really need to see the output of something like curl -v -s http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz > /dev/null to get headers and the HTTP response from within a container. My guess is a temporary issue with a Fastly POP or a change in POP IP address that hasn't yet properly timed out in Docker DNS proxy yet.
@andyshinn I'll see if I can provide you with the output. However, I believe the issue is in Docker Engine, and possibly in DNS caching. My Host Machine was on the corporate network 1, but when we switched it to corporate network 2, the issue was gone. After speaking to networking people, they weren't able to tell how the difference between networks could have affected the Docker Build. It points to a combination of DNS and Docker Engine. However, we ruled out the DNS, because on both networks we were able to do curl from the host and download the .gz files.
I checked my docker config and its DNS is set to 8.8.8.8 and 8.8.4.4. From both networks these IPs are reachable.
I think it is related to #279 which has more history. I think maybe when Fastly changes server IPs they may be redirecting with a 301, which causes apk to choke. But this is why I want to output to see where it is redirecting. I'm not sure if it comes from Fastly or a broken upstream server.
So I'm having dns issues, but my results are different. My stuff just hangs at getting packages. https://www.whatsmydns.net/#A/mirrors.aliyun.com
This is the mirror mine tries to use.
curl -vvvv -s http://mirrors.aliyun.com/alpine/v3.5/main/x86_64/APKINDEX.tar.gz > /dev/null
* Trying 124.47.11.88...
* Connected to mirrors.aliyun.com (124.47.11.88) port 80 (#0)
> GET /alpine/v3.5/main/x86_64/APKINDEX.tar.gz HTTP/1.1
> Host: mirrors.aliyun.com
> User-Agent: curl/7.47.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
It will just hang here indefinitely in docker (at least for an hour+) and the curl times out. This is from my laptop:
recipes: curl -vvvv -s http://mirrors.aliyun.com/alpine/v3.5/main/x86_64/APKINDEX.tar.gz > /dev/null
* Trying 219.147.157.84...
* TCP_NODELAY set
* Connected to mirrors.aliyun.com (219.147.157.84) port 80 (#0)
> GET /alpine/v3.5/main/x86_64/APKINDEX.tar.gz HTTP/1.1
> Host: mirrors.aliyun.com
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Tengine
< Content-Type: application/octet-stream
< Content-Length: 732690
< Connection: keep-alive
< Date: Wed, 27 Sep 2017 13:45:47 GMT
< Last-Modified: Thu, 21 Sep 2017 10:13:30 GMT
< ETag: "59c390ca-b2e12"
< Expires: Wed, 27 Sep 2017 13:50:47 GMT
< Cache-Control: max-age=300
< Accept-Ranges: bytes
< Via: cache14.l2nu29[36,304-0,H], cache32.l2nu29[38,0], cache2.cn547[0,200-0,H], cache6.cn547[1,0]
< Age: 18
< X-Cache: HIT TCP_MEM_HIT dirn:3:81977809 mlen:-1
< X-Swift-SaveTime: Wed, 27 Sep 2017 13:45:48 GMT
< X-Swift-CacheTime: 120
< Timing-Allow-Origin: *
< EagleId: db939d4615065199657165411e
<
{ [12630 bytes data]
* Curl_http_done: called premature == 0
* Connection #0 to host mirrors.aliyun.com left intact
@andyshinn this may be a different issue. But it's DNS related, and works one place but not the other. This is from my system, but if you need it from inside docker, I can provide it.
Have you tried another mirror? From that debug output, the IP address resolved from Docker and from the laptop are different, so it could definitely be an issue with just that host. Have you actually tried using the same IP on your laptop that Docker resolves to (something like curl -v -H 'Host: mirrors.aliyun.com' http://124.47.11.88/alpine/v3.5/main/x86_64/APKINDEX.tar.gz > /dev/null)?
Other mirrors do work. I suppose I have to go dig and see if they hard coded a mirror or something. It's odd though that it breaks apk so hard. I suppose I would have thought it to be more resilient
Its in general, related to DNS/apk fragility, but I think perhaps a different issue then op:
https://github.com/laradock/laradock/issues/749#issuecomment-293296687
I was seeing this error on my Drone build server, running on Kubernetes. By default it users "Docker-in-Docker" (dind) for builds. The error went away after I disabled dind on the Drone server.
@matkam - Could you detail your solution ? We seem to having the same issue with our drone server.
@Zest-nikunj sure. My Drone server was deployed by a Helm chart to a Kubernetes cluster: https://github.com/kubernetes/charts/tree/master/incubator/drone
I just redeployed it, setting dind.enabled to false. I'd imagine if you're not using Helm or Kubernetes, you'd have to mount the host's docker socket on the drone agent's container: https://github.com/kubernetes/charts/blob/master/incubator/drone/templates/deployment-agent.yaml#L52
Thanks @matkam. We are not on Kubernetes and the settings seem correct but we continue to have the issue. We will investigate further. If needed, I shall come back to you. Thanks again.
Same problem, just kept on re-trying, and 3rd time it went thru. seems like there is a bad server which the load balancer hits.
same problem here . I tried many many times ... it is not a bad server ... it seams a bad code line , a bug
I got same. I tried to change dns on docker with no luck. I'm on Ubuntu 18.04, my colleague can run successfully exactly same build in the same network on Arch
Hit the same problem (again) on Windows + Docker and tried many suggested (in #279) solutions, involving changing apk settings in Dockerfile and such. but nothing helped. What did help finally was to restart Docker and then apk just pulled the APKINDEX like nobody's business.
I get this as well on my workstation using Docker for Windows. Using a mirror did not help.
I had this error with Gitlab, i just change docker build command to run with --network host option and it works!
sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf helped on Ubuntu 16 build agent to make CI jobs build alpine based docker images again as per https://github.com/gliderlabs/docker-alpine/issues/279#issuecomment-411982620
@EvgeniGordeev That solution worked for me too, running Windows 10 host + Ubuntu 18.04 with Vagrant and alpinelinux in a docker container in the Ubuntu.
@andyshinn , RE: https://github.com/gliderlabs/docker-alpine/issues/334#issuecomment-330096922
It was a DNS error for me. By setting /etc/docker/daemon.json with,
{
"dns": ["8.8.8.8"]
}
and then restarting docker with,
sudo service docker restart
I was able to build images again.
Just for reference, I got a similar error, but in my case it was a problem with nftables:
---> Running in 2522cf9dc903
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)
ERROR: unsatisfiable constraints:
Stopping nftables and restarting docker will give the container internet access.
Since there are times when the bridge is disturbing, it is successful if you execute it as host.
docker build -t hoge:latest . --network=host
Run docker build command to run with --network=host option and it works
any idea how to add this workaround using docker-compose ? the --network flag is not available there
I just ran into this, everything had been working fine on:
Windows 10 host with Hyper-V Ubuntu 18.04 Guest Docker in the Ubuntu 18.04
Suddenly apk and go mod download started failing. The APK issue was the temporary error. Did the fix suggested in this comment https://github.com/gliderlabs/docker-alpine/issues/334#issuecomment-450598069 and everything was working again. DNS Was working fine on the guest and the host, only when building the docker containers was I having issues.
same error
Ubuntu in VMware Docker in the Ubuntu 18.04
change virtual machine's network setting to NAT mode, and it works
