docker-image-resource icon indicating copy to clipboard operation
docker-image-resource copied to clipboard

Published 20 hours ago • verified publisher icon concourse

Cannot pull from dockerhub within docker-compose setup

Open chris-aeviator opened this issue 7 years ago • 5 comments

When trying to add a resource from dockerhub (compare hello world tutorial) task will fail when using the the setup through official docker-compose.yml.

When changing the docker registry mirror to e.g. http://quay.io the tasi will perfectly pull the image.

This issue is basically blocking all work I'm planing to do with concourse and I'd be really thankful for a workaround?!

Config dockerhub

- name: rsync-resource
  type: docker-image
  source:
      repository: mrsixw/concourse-rsync-resource
      tag: latest

Output dockerhub:

sha256:6f1e44344622359b89797538916bea5e27566fea5f79e3fbd8bc8e8df50341e6: Pulling from mrsixw/concourse-rsync-resource
b0348695caf2: Pulling fs layer
204ea97fe778: Pulling fs layer
416b5a2739d4: Pulling fs layer
error pulling image configuration: Get https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/5a/5a551f16b2a80da331e0b7eb4d908892a1424fc1cd83f86709f02d2b02cc5ee9/data?Expires=1503433983&Signature=UFBxOqGoi9QkDYP8omqklN4CbJrdpZZLsYftz93ur245CzIRCSAee5oD6DIVAfD95XgyMP4dpxx0OfJQ2NCuHRaK7pexQdbXDwk7nbumYFQ6XixgrVtMUk1zn9gWLRPpjV-e09dtpVIWTmsZbZ47kOEpq5EmafM9CV0eCZ3AqKU_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q: dial tcp: lookup dseasb33srnrn.cloudfront.net on 172.18.0.4:53: read udp 10.254.0.90:48799->172.18.0.4:53: i/o timeout

Pulling mrsixw/concourse-rsync-resource@sha256:6f1e44344622359b89797538916bea5e27566fea5f79e3fbd8bc8e8df50341e6 (attempt 2 of 3)...
sha256:6f1e44344622359b89797538916bea5e27566fea5f79e3fbd8bc8e8df50341e6: Pulling from mrsixw/concourse-rsync-resource
b0348695caf2: Pulling fs layer
204ea97fe778: Pulling fs layer
416b5a2739d4: Pulling fs layer
error pulling image configuration: Get https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/5a/5a551f16b2a80da331e0b7eb4d908892a1424fc1cd83f86709f02d2b02cc5ee9/data?Expires=1503433995&Signature=GXhG~-SE2oy~cSoL753mLk85WCIfgyFKi47h~s1DXaA4hPoLm7VPaCwSR7UVHXlyodJSYnP4kPDe-QGn~mKMuC2CJNm2dpesLJQKo-VQOrRwq17XaH6hZe98Fz4lRgk4OtOGLbWoHkD3hcLlmx1wB7jrE2xHq2xaRMpBjHZM2ek_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q: dial tcp: lookup dseasb33srnrn.cloudfront.net on 172.18.0.4:53: read udp 10.254.0.90:58077->172.18.0.4:53: i/o timeout

Pulling mrsixw/concourse-rsync-resource@sha256:6f1e44344622359b89797538916bea5e27566fea5f79e3fbd8bc8e8df50341e6 (attempt 3 of 3)...
sha256:6f1e44344622359b89797538916bea5e27566fea5f79e3fbd8bc8e8df50341e6: Pulling from mrsixw/concourse-rsync-resource
b0348695caf2: Pulling fs layer
204ea97fe778: Pulling fs layer
416b5a2739d4: Pulling fs layer
error pulling image configuration: Get https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/5a/5a551f16b2a80da331e0b7eb4d908892a1424fc1cd83f86709f02d2b02cc5ee9/data?Expires=1503434006&Signature=RSbEK-pQ7PLLHy6VlhsSjD0bmm83vaieanITGUWElZT0AW7Hqsp3cXQQFPBRp3fXkosIBDYdMXMkM8wyfreGT1hyQLPrRIyPCB0z-uUW0k1vSwloHq1zxWe~E5mFBWS7yY5zJQNVuz4BJSbMQj271uwtftHiv-WffJXvCVQwO0Y_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q: dial tcp: lookup dseasb33srnrn.cloudfront.net on 172.18.0.4:53: read udp 10.254.0.90:53558->172.18.0.4:53: i/o timeout

Config registry mirror

- task: web-app-tests
    config:
      platform: linux
      image_resource:
          type: docker-image
          source:
            registry_mirror: http://quay.io
            repository: jgwmaxwell/debian
            tag: latest

output registry mirror

Pulling jgwmaxwell/debian@sha256:9d87f169eb52534739c17d44194c8ab5d4131ad52f4b1e1f2fd3c53a5c55ac19...
sha256:9d87f169eb52534739c17d44194c8ab5d4131ad52f4b1e1f2fd3c53a5c55ac19: Pulling from jgwmaxwell/debian

downloads fine

chris-aeviator avatar Aug 22 '17 20:08 chris-aeviator

as a quick solution I am now pulling the needed Docker images to the host machine and running my own docker-registry in the docker network where I push the images from the host maschine.

That way concourse is able to find the images when adding like so:

resource_types:
- name: rsync-resource
  type: docker-image
  source:
      insecure_registries: ["172.18.0.5:5000"]
      repository: 172.18.0.5:5000/concourse-rsync
      tag: master

chris-aeviator avatar Aug 23 '17 11:08 chris-aeviator

@chris-aeviator, it looks like your local DNS is not able to lookup dockerhub.

Can you run dig hub.docker.com on your host machine, in a container, and on a task?

jtarchie avatar Aug 25 '17 14:08 jtarchie

Exact same problem here, and also switched to the workaround as described by Chris. I can run dig on the host machine and it resolves:

; <<>> DiG 9.10.3-P4-Ubuntu <<>> hub.docker.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41690
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;hub.docker.com.			IN	A

;; ANSWER SECTION:
hub.docker.com.		20	IN	CNAME	elb-default.us-east-1.aws.dckr.io.
elb-default.us-east-1.aws.dckr.io. 139 IN CNAME	us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com.
us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com.	39 IN A	52.73.244.162
us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com.	39 IN A	52.55.168.191
us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com.	39 IN A	34.201.187.190

;; AUTHORITY SECTION:
us-east-1.elb.amazonaws.com. 1364 IN	NS	ns-235.awsdns-29.com.
us-east-1.elb.amazonaws.com. 1364 IN	NS	ns-934.awsdns-52.net.
us-east-1.elb.amazonaws.com. 1364 IN	NS	ns-1119.awsdns-11.org.
us-east-1.elb.amazonaws.com. 1364 IN	NS	ns-1793.awsdns-32.co.uk.

;; Query time: 2 msec
;; SERVER: 213.133.98.98#53(213.133.98.98)
;; WHEN: Sat Sep 09 09:03:58 CEST 2017
;; MSG SIZE  rcvd: 356

In the concourse worker container this also works but I had to install dnsutils first. Did a docker exec to test this, btw.

On a task it fails with the above error message. Task def is:

    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: ubuntu
          tag: 16.04
      run:
        path: bash
        args:
        - -exc
        - |
          dig hub.docker.com

UFW and fail2ban were stopped/disabled in all test cases.

Cheers, Erwin

eembsen avatar Sep 09 '17 07:09 eembsen

I've got exactly the same symptoms described by @eembsen. Any reasonable explanation on this? Why would the DNS problems occur only for the containers spawned by our tasks?

daniellavoie avatar Dec 04 '17 02:12 daniellavoie

We managed to get around it by iptables DNAT-ing the DNS packets to a different DNS server than the one hosted by the Concourse Worker process. Any idea why the DNS on the Worker does not work with the resource trying to upload the image, but yet works from the hijacked container?

dvianello avatar Jun 07 '18 16:06 dvianello