Pushing local images to clair built from recent docker hub images results in clairctl 404 / clair 400

[ninjaMog]

I have an issue when pushing local images I've fetched from docker hub, or local images I've built based on recent docker hub images:

14:41 $ ./clairctl analyze -l --log-level debug nginx:latest
2017-11-23 14:41:32.266687 D | config: Using config file: /Users/anon/.go/src/github.com/jgsqware/clairctl/clairctl.yml
2017-11-23 14:41:32.266958 D | dockercli: docker image to save: nginx:latest
2017-11-23 14:41:32.266967 D | dockercli: saving in: /tmp/clairctl/nginx/blobs
2017-11-23 14:41:33.815072 I | config: retrieving interface for local IP
2017-11-23 14:41:33.815093 D | config: interface provided, looking for utun1
2017-11-23 14:41:33.815363 D | server: Update local server port from "0" to "54632"
2017-11-23 14:41:33.815372 I | server: Starting Server on 10.1.101.123:54632
2017-11-23 14:41:33.821723 I | config: retrieving interface for local IP
2017-11-23 14:41:33.821760 D | config: interface provided, looking for utun1
2017-11-23 14:41:33.821892 I | clair: using http://10.1.101.123:54632/local as local url
2017-11-23 14:41:33.821908 I | clair: Pushing Layer 1/3 [912743f98e2a]
2017-11-23 14:41:33.822000 D | clair: Saving 912743f98e2abc5b51dd54d3cc02af68c037580ea1692056ee2eeea8625858a1[https://registry-1.docker.io/v2]
2017-11-23 14:41:34.226044 D | server: Received request for /docker.io/library/nginx/blobs/912743f98e2abc5b51dd54d3cc02af68c037580ea1692056ee2eeea8625858a1/layer.tar
2017-11-23 14:41:34.323759 I | clair: adding layer 1/3 [912743f98e2a]: receiving http error: 400
client quit unexpectedly
2017-11-23 14:41:34.323799 C | cmd: pushing image "nginx:latest": receiving http error: 400

(I added additional debug to clairctl version 1.2.8 to show the path for the received requests as clair / clairctl wasn't being particularly verbose)

You can see from this that the request includes a docker.io/library prefix, but the files in /tmp/clairctl were not prefixed:

/tmp/clairctl
15:16 $ ls -al
total 0
drwxr-xr-x  3 user  wheel  102 23 Nov 15:16 .
drwxrwxrwt  8 root       wheel  272 23 Nov 15:16 ..
drwxr-xr-x  3 user  wheel  102 23 Nov 15:16 nginx

I can workaround this by making a symlink /tmp/clairctl/docker.io/library -> /tmp/clairctl/ , the scan works fine then.

I wonder if this is caused by the recent update to the docker hub manifest files? (https://github.com/jgsqware/clairctl/issues/93)

[jdel]

Hi, thanks for raising this issue. This looks like some other issue we already have with local paths.

And unfortunately, I haven't had time to investigate this.

[acdha]

I just confirmed that the workaround @ninjaMog mentioned worked on a clean install

[toddysm]

I have the same issue but the strange thing is that it only happens when the Clair server is not on the local machine. For example, if the Clair server is remote, I receive the following in Debug log:

$ clairctl analyze --local busybox --config /root/Development/clair/clairctl.yml  --log-level Debug
2017-12-04 18:04:51.897364 D | config: Using config file: /root/Development/clair/clairctl.yml
2017-12-04 18:04:51.897609 D | dockercli: docker image to save: busybox:latest
2017-12-04 18:04:51.897617 D | dockercli: saving in: /tmp/clairctl/busybox/blobs
2017-12-04 18:04:51.923934 I | config: retrieving interface for local IP
2017-12-04 18:04:51.923961 D | config: no interface provided, looking for docker0
2017-12-04 18:04:51.924439 D | server: Update local server port from "0" to "43746"
2017-12-04 18:04:51.924453 I | server: Starting Server on 172.17.0.1:43746
2017-12-04 18:04:51.929422 I | config: retrieving interface for local IP
2017-12-04 18:04:51.929434 D | config: no interface provided, looking for docker0
2017-12-04 18:04:51.929662 I | clair: using http://172.17.0.1:43746/local as local url
2017-12-04 18:04:51.929674 I | clair: Pushing Layer 1/1 [036a82c6d65f]
2017-12-04 18:04:51.929749 D | clair: Saving 036a82c6d65f2fa43a13599661490be3fca1c3d6790814668d4e8c0213153b12[https:///v2]
2017-12-04 18:04:52.017592 I | clair: adding layer 1/1 [036a82c6d65f]: receiving http error: 400
client quit unexpectedly
2017-12-04 18:04:52.017621 C | cmd: pushing image "busybox:latest": receiving http error: 400

If I change the configuration to point to a local Clair instance, everything is fine and the scan works:

2017-12-04 18:06:30.508243 D | config: Using config file: /root/Development/clair/clairctl.yml
2017-12-04 18:06:30.508698 D | dockercli: docker image to save: busybox:latest
2017-12-04 18:06:30.508707 D | dockercli: saving in: /tmp/clairctl/busybox/blobs
2017-12-04 18:06:30.535114 I | config: retrieving interface for local IP
2017-12-04 18:06:30.535158 D | config: no interface provided, looking for docker0
2017-12-04 18:06:30.535611 D | server: Update local server port from "0" to "39788"
2017-12-04 18:06:30.535625 I | server: Starting Server on 172.17.0.1:39788
2017-12-04 18:06:30.540607 I | config: retrieving interface for local IP
2017-12-04 18:06:30.540619 D | config: no interface provided, looking for docker0
2017-12-04 18:06:30.540820 I | clair: using http://172.17.0.1:39788/local as local url
2017-12-04 18:06:30.540830 I | clair: Pushing Layer 1/1 [036a82c6d65f]
2017-12-04 18:06:30.540909 D | clair: Saving 036a82c6d65f2fa43a13599661490be3fca1c3d6790814668d4e8c0213153b12[https:///v2]
2017-12-04 18:06:30.545216 I | config: retrieving interface for local IP
2017-12-04 18:06:30.545228 D | config: no interface provided, looking for docker0
2017-12-04 18:06:30.545423 I | clair: using http://172.17.0.1:39788/local as local url
2017-12-04 18:06:30.549019 I | clair: analysing layer [036a82c6d65f] 1/1
2017-12-04 18:06:30.549036 D | cmd: Using priority filters:

Image: /busybox:latest

 Unknown: 0
 Negligible: 0
 Low: 0
 Medium: 0
 High: 0
 Critical: 0
 Defcon1: 0

@ninjaMog I am not sure what you mean by symlinking /tmp/clairctl/docker.io/library -> /tmp/clairctl/ - doesn't this create a circular reference?

[vchan2002]

I have the same issue also... even after pulling the image locally, then trying to push it as a local image, I still get this error.

2018-01-20 00:28:40.790971 I | config: retrieving interface for local IP 2018-01-20 00:28:40.790999 D | config: no interface provided, looking for docker0 2018-01-20 00:28:40.791205 I | clair: using http://172.17.0.1:35839/local as local url 2018-01-20 00:28:40.791224 I | clair: Pushing Layer 1/13 [13d1f98ad237] 2018-01-20 00:28:40.791371 D | clair: Saving 13d1f98ad237ccc31732c44481c0f8af72481ffcdb996fb976aa6ebb613c4199[https:///v2] 2018-01-20 00:28:40.793096 I | clair: adding layer 1/13 [13d1f98ad237]: receiving http error: 404 client quit unexpectedly

[lukeb2e]

The issue is resolved in https://github.com/jgsqware/clairctl/pull/112

It is related to a false temp save path for the blobs

As a current workaround you need to compile your own binary or wait until the pull request is merged