docker-compose-buildkite-plugin
docker-compose-buildkite-plugin copied to clipboard
Published
20 hours ago •
buildkite-plugins
buildkite-plugins
Docker credentials not recognized from agent, but recognized outside agent
Heya,
I have a Windows VM on Google Cloud setup to run BuildKite agents. I've been running the buildkite docker plugin for a while and it has been working. Now tried to use the docker compose plugin, and also tried to pull an image from the Google Cloud Registry.
That last bit fails: https://buildkite.com/angular/angular/builds/1649#_
Building services windows-test | 2m 13s
-- | --
| [2019-01-05T19:39:29Z] $ docker-compose -f .buildkite/dockerfiles/docker-compose.yml -p buildkite0a4f2db409064e028975693efdd2fe69 -f docker-compose.buildkite-1649-override.yml build --pull --build-arg target=gcr.io/internal-200822/angular-windows:master windows-test
| [2019-01-05T19:39:30Z] Building windows-test
| [2019-01-05T19:41:40Z] Step 1/9 : ARG target=filipesilva/node-bazel-windows:0.0.2
| [2019-01-05T19:41:40Z] Step 2/9 : FROM $target
| [2019-01-05T19:41:41Z] Service 'windows-test' failed to build: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
| [2019-01-05T19:41:41Z] 🚨 Error: The command exited with status 1
At this point I have already configured the credentials as documented in https://cloud.google.com/container-registry/docs/advanced-authentication, using the gcloud as a Docker credential helper method. It seems to configure some credential helpers automatically:
PS C:\Users\angular\.docker> cat .\config.json
{
"credHelpers": {
"gcr.io": "gcloud",
"us.gcr.io": "gcloud",
"eu.gcr.io": "gcloud",
"asia.gcr.io": "gcloud",
"staging-k8s.gcr.io": "gcloud",
"marketplace.gcr.io": "gcloud"
}
}
The interesting thing is that if I just go into the same directory as the agent was, and run the same command, it will have no auth problems:
PS C:\buildkite-agent\builds\gce-buildkite-windows-2-3\angular\angular> docker-compose -f .buildkite/dockerfiles/docker-
compose.yml -p buildkite0a4f2db409064e028975693efdd2fe69 -f docker-compose.buildkite-1649-override.yml build --pull --bu
ild-arg target=gcr.io/internal-200822/angular-windows:master windows-test
Building windows-test
Step 1/9 : ARG target=filipesilva/node-bazel-windows:0.0.2
Step 2/9 : FROM $target
master: Pulling from internal-200822/angular-windows
Digest: sha256:feb730ea3a7c0b60f7315562b4d3d234215e762687467974a33d29eb0ae4bbf5
Status: Image is up to date for gcr.io/internal-200822/angular-windows:master
---> dfb9a2da8124
Step 3/9 : WORKDIR /src
---> Running in f8d3928aa948
Removing intermediate container f8d3928aa948
---> 78576cac0021
Step 4/9 : COPY ./ /src
etc etc
Do you have any idea of what might be happening? Perhaps there is some override for credential helpers, or for the default docker config.json file?
I also tried running the it with the verbose flag in the config: https://buildkite.com/angular/angular/builds/1652#fa3ad162-860e-49a0-8da1-f3cf31faf603
[2019-01-05T20:14:01Z] $ docker-compose --verbose -f .buildkite/dockerfiles/docker-compose.yml -p buildkitefa3ad162860e49a08da1f3cf31faf603 -f docker-compose.buildkite-1652-override.yml build --pull --build-arg target=gcr.io/internal-200822/angular-windows:master windows-test
--
| [2019-01-05T20:14:02Z] compose.config.config.find: Using configuration files: .\.buildkite/dockerfiles/docker-compose.yml,.\docker-compose.buildkite-1652-override.yml
| [2019-01-05T20:14:02Z] docker.utils.config.find_config_file: Trying paths: ['C:\\Windows\\system32\\config\\systemprofile\\.docker\\config.json', 'C:\\Windows\\system32\\config\\systemprofile\\.dockercfg']
| [2019-01-05T20:14:02Z] docker.utils.config.find_config_file: No config file found
| [2019-01-05T20:14:02Z] docker.utils.config.find_config_file: Trying paths: ['C:\\Windows\\system32\\config\\systemprofile\\.docker\\config.json', 'C:\\Windows\\system32\\config\\systemprofile\\.dockercfg']
| [2019-01-05T20:14:02Z] docker.utils.config.find_config_file: No config file found
| [2019-01-05T20:14:02Z] urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/version HTTP/1.1" 200 587
| [2019-01-05T20:14:02Z] compose.cli.command.get_client: docker-compose version 1.23.2, build 1110ad01
| [2019-01-05T20:14:02Z] docker-py version: 3.6.0
| [2019-01-05T20:14:02Z] CPython version: 3.6.6
| [2019-01-05T20:14:02Z] OpenSSL version: OpenSSL 1.0.2o 27 Mar 2018
| [2019-01-05T20:14:02Z] compose.cli.command.get_client: Docker base_url: http+docker://localnpipe
| [2019-01-05T20:14:02Z] compose.cli.command.get_client: Docker version: Platform={'Name': ''}, Components=[{'Name': 'Engine', 'Version': '18.09.0', 'Details': {'ApiVersion': '1.39', 'Arch': 'amd64', 'BuildTime': '11/07/2018 00:24:12', 'Experimental': 'false', 'GitCommit': '33a45cd0a2', 'GoVersion': 'go1.10.3', 'KernelVersion': '10.0 17134 (17134.1.amd64fre.rs4_release.180410-1804)', 'MinAPIVersion': '1.24', 'Os': 'windows'}}], Version=18.09.0, ApiVersion=1.39, MinAPIVersion=1.24, GitCommit=33a45cd0a2, GoVersion=go1.10.3, Os=windows, Arch=amd64, KernelVersion=10.0 17134 (17134.1.amd64fre.rs4_release.180410-1804), BuildTime=11/07/2018 00:24:12
| [2019-01-05T20:14:02Z] compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('buildkitefa3ad162860e49a08da1f3cf31faf603_default')
| [2019-01-05T20:14:02Z] urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/networks/buildkitefa3ad162860e49a08da1f3cf31faf603_default HTTP/1.1" 404 82
| [2019-01-05T20:14:02Z] compose.service.build: Building windows-test
| [2019-01-05T20:14:02Z] compose.cli.verbose_proxy.proxy_callable: docker build <- (path='\\\\?\\C:\\buildkite-agent\\builds\\gce-buildkite-windows-2-4\\angular\\angular', tag='angular-windows-test-build-1652', rm=True, forcerm=False, pull=True, nocache=False, dockerfile='./.buildkite/dockerfiles/windows-test.Dockerfile', cache_from=None, labels=None, buildargs={'target': 'gcr.io/internal-200822/angular-windows:master'}, network_mode=None, target=None, shmsize=None, extra_hosts=None, container_limits={'memory': None}, gzip=False, isolation=None, platform=None)
| [2019-01-05T20:14:44Z] docker.api.build._set_auth_headers: Looking for auth config
| [2019-01-05T20:14:44Z] docker.api.build._set_auth_headers: No auth config in memory - loading from filesystem
| [2019-01-05T20:14:44Z] docker.utils.config.find_config_file: Trying paths: ['C:\\Windows\\system32\\config\\systemprofile\\.docker\\config.json', 'C:\\Windows\\system32\\config\\systemprofile\\.dockercfg']
| [2019-01-05T20:14:44Z] docker.utils.config.find_config_file: No config file found
| [2019-01-05T20:14:44Z] docker.api.build._set_auth_headers: No auth config found
| [2019-01-05T20:14:54Z] urllib3.connectionpool._make_request: http://localhost:None "POST /v1.25/build?t=angular-windows-test-build-1652&q=False&nocache=False&rm=True&forcerm=False&pull=True&dockerfile=.%2F.buildkite%2Fdockerfiles%2Fwindows-test.Dockerfile&buildargs=%7B%22target%22%3A+%22gcr.io%2Finternal-200822%2Fangular-windows%3Amaster%22%7D HTTP/1.1" 200 None
| [2019-01-05T20:14:54Z] compose.cli.verbose_proxy.proxy_callable: docker build -> <generator object APIClient._stream_helper at 0x00000236C68408E0>
| [2019-01-05T20:14:54Z] Step 1/9 : ARG target=filipesilva/node-bazel-windows:0.0.2
| [2019-01-05T20:14:54Z] Step 2/9 : FROM $target
| [2019-01-05T20:14:55Z] compose.cli.main.main: Service 'windows-test' failed to build: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
| [2019-01-05T20:14:56Z] 🚨 Error: The command exited with status 1
And manually in the the directory:
PS C:\buildkite-agent\builds\gce-buildkite-windows-2-3\angular\angular> docker-compose -f .buildkite/dockerfiles/docker-
compose.yml -p buildkite0a4f2db409064e028975693efdd2fe69 -f docker-compose.buildkite-1649-override.yml --verbose build -
-pull --build-arg target=gcr.io/internal-200822/angular-windows:master windows-test
compose.config.config.find: Using configuration files: .\.buildkite/dockerfiles/docker-compose.yml,.\docker-compose.buildkite-1649-override.yml
docker.utils.config.find_config_file: Trying paths: ['C:\\Users\\angular\\.docker\\config.json', 'C:\\Users\\angular\\.dockercfg']
docker.utils.config.find_config_file: Found file at path: C:\Users\angular\.docker\config.json
docker.auth.load_config: Found 'credHelpers' section
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/version HTTP/1.1" 200 587
compose.cli.command.get_client: docker-compose version 1.23.2, build 1110ad01
docker-py version: 3.6.0
CPython version: 3.6.6
OpenSSL version: OpenSSL 1.0.2o 27 Mar 2018
compose.cli.command.get_client: Docker base_url: http+docker://localnpipe
compose.cli.command.get_client: Docker version: Platform={'Name': ''}, Components=[{'Name': 'Engine', 'Version': '18.09.0', 'Details': {'ApiVersion': '1.39', 'Arch': 'amd64', 'BuildTime': '11/07/2018 00:24:12', 'Experimental': 'false', 'GitCommit': '33a45cd0a2', 'GoVersion': 'go1.10.3', 'KernelVersion': '10.0 17134 (17134.1.amd64fre.rs4_release.180410-1804)', 'MinAPIVersion': '1.24', 'Os': 'windows'}}], Version=18.09.0, ApiVersion=1.39, MinAPIVersion=1.24, GitCommit=33a45cd0a2, GoVersion=go1.10.3, Os=windows, Arch=amd64, KernelVersion=10.0 17134 (17134.1.amd64fre.rs4_release.180410-1804), BuildTime=11/07/2018 00:24:12
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('buildkite0a4f2db409064e028975693efdd2fe69_default')
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/networks/buildkite0a4f2db409064e028975693efdd2fe69_default HTTP/1.1" 404 82
compose.service.build: Building windows-test
compose.cli.verbose_proxy.proxy_callable: docker build <- (path='\\\\?\\C:\\buildkite-agent\\builds\\gce-buildkite-windows-2-3\\angular\\angular', tag='angular-windows-test-build-1649', rm=True, forcerm=False, pull=True, nocache=False, dockerfile='./.buildkite/dockerfiles/windows-test.Dockerfile', cache_from=None, labels=None, buildargs={'target': 'gcr.io/internal-200822/angular-windows:master'}, network_mode=None, target=None, shmsize=None, extra_hosts=None, container_limits={'memory': None}, gzip=False, isolation=None, platform=None)
docker.api.build._set_auth_headers: Looking for auth config
docker.auth.resolve_authconfig: Using credentials store "gcloud"
docker.auth._resolve_authconfig_credstore: Looking for auth entry for 'gcr.io'
docker.auth.resolve_authconfig: Using credentials store "gcloud"
docker.auth._resolve_authconfig_credstore: Looking for auth entry for 'us.gcr.io'
docker.auth.resolve_authconfig: Using credentials store "gcloud"
docker.auth._resolve_authconfig_credstore: Looking for auth entry for 'eu.gcr.io'
docker.auth.resolve_authconfig: Using credentials store "gcloud"
docker.auth._resolve_authconfig_credstore: Looking for auth entry for 'asia.gcr.io'
docker.auth.resolve_authconfig: Using credentials store "gcloud"
docker.auth._resolve_authconfig_credstore: Looking for auth entry for 'staging-k8s.gcr.io'
docker.auth.resolve_authconfig: Using credentials store "gcloud"
docker.auth._resolve_authconfig_credstore: Looking for auth entry for 'marketplace.gcr.io'
docker.api.build._set_auth_headers: Sending auth config ('gcr.io', 'us.gcr.io', 'eu.gcr.io', 'asia.gcr.io', 'staging-k8s.gcr.io', 'marketplace.gcr.io')
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.25/build?t=angular-windows-test-build-1649&q=False&nocache=False&rm=True&forcerm=False&pull=True&dockerfile=.%2F.buildkite%2Fdockerfiles%2Fwindows-test.Dockerfile&buildargs=%7B%22target%22%3A+%22gcr.io%2Finternal-200822%2Fangular-windows%3Amaster%22%7D HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker build -> <generator object APIClient._stream_helper at 0x000001A22195E728>
Step 1/9 : ARG target=filipesilva/node-bazel-windows:0.0.2
Step 2/9 : FROM $target
master: Pulling from internal-200822/angular-windows
Digest: sha256:feb730ea3a7c0b60f7315562b4d3d234215e762687467974a33d29eb0ae4bbf5
Status: Image is up to date for gcr.io/internal-200822/angular-windows:master
---> dfb9a2da8124
Step 3/9 : WORKDIR /src
The important bit seems to be that the plugin cannot find a config file:
| [2019-01-05T20:14:02Z] docker.utils.config.find_config_file: Trying paths: ['C:\\Windows\\system32\\config\\systemprofile\\.docker\\config.json', 'C:\\Windows\\system32\\config\\systemprofile\\.dockercfg']
| [2019-01-05T20:14:02Z] docker.utils.config.find_config_file: No config file found
| [2019-01-05T20:14:02Z] docker.utils.config.find_config_file: Trying paths: ['C:\\Windows\\system32\\config\\systemprofile\\.docker\\config.json', 'C:\\Windows\\system32\\config\\systemprofile\\.dockercfg']
| [2019-01-05T20:14:02Z] docker.utils.config.find_config_file: No config file found
But that the manual call can:
docker.utils.config.find_config_file: Trying paths: ['C:\\Users\\angular\\.docker\\config.json', 'C:\\Users\\angular\\.dockercfg']
docker.utils.config.find_config_file: Found file at path: C:\Users\angular\.docker\config.json
Looking at those specific logs it looks related to the current docker user.
These agents are setup as a service using NSSM, as instructed in https://buildkite.com/docs/agent/v3/windows#running-as-a-service. It looks like NSSM uses the local system account by default.
I copied the docker config there via:
Copy-Item C:\Users\angular\.docker -Destination C:\Windows\System32\config\systemprofile\ -Recurse
It seems to work!
Not sure if there's any action on your side, so perhaps this should be closed. If there's a way to pass the docker config file (not the docker-compose file) I suppose that would be good.
Sorry, I've been slow to respond to this one. At a minimum we should document that nuance in those instructions, and probably suggest running the service as the local user vs a local system account.
Looks like the issue was indeed solved and was due to a configuration discrepancy between the user the agent was running as. Closing this one as there isn't much we can do here but feel free to re-open if you have any suggestions
