`DOCKER_AUTH_CONFIG` takes precedence over `docker login`

[fabriceclementz]

Description

First of all, apologies if this is not the appropriate place to raise this — please feel free to redirect me if this should be handled elsewhere.

Context

Since the merge of PR #6008, we're experiencing issues when pushing to a private registry from a GitLab CI job using the docker:dind image.

Our GitLab CI pipelines pull images from a private registry using a read-only service account. To support this, we initialize the DOCKER_AUTH_CONFIG environment variable with a base64-encoded JSON config containing the read-only credentials (following the principle of least privilege). This is mandatory to allow the GitLab Runner to pull our job image.

Later in the pipeline, we perform a docker login with another service account that has write permissions, in order to push new images. The login correctly updates the ~/.docker/config.json file.

Issue

After the change introduced by this PR #6008, it seems that the DOCKER_AUTH_CONFIG environment variable continues to take precedence, even after a successful docker login. As a result, docker push fails with permission denied errors because it is still using the read-only credentials from DOCKER_AUTH_CONFIG.

Furthermore, there is no message or warning from the Docker CLI indicating that DOCKER_AUTH_CONFIG is being used in preference to the updated login credentials. This makes it especially difficult to diagnose the root cause, as one would expect the docker login command to override or be honored for subsequent operations.

Previously, docker login would override the current auth context, allowing the push to succeed using the updated credentials.

Maybe I misunderstood this change and we could do otherwise? Or maybe this is a bug introduced with this PR. Thanks for your help.

Reproduce

1. Use docker:dind in GitLab CI.
2. Set DOCKER_AUTH_CONFIG with read-only credentials to access a private registry.
3. Run docker login private-registry.xxx.com with credentials that have write access.
4. Attempt to run docker push private-registry.xxx.com/repo/image:tag.

unauthorized: unauthorized to access repository: repo/image, action: push: unauthorized to access repository: repo/image, action: push

Expected behavior

docker push should succeed using the credentials updated via docker login.

docker version

Client:
 Version:    28.3.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.25.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.37.3
    Path:     /usr/local/libexec/docker/cli-plugins/docker-compose
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 28.3.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.6-0-ge89a299
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.16.14-1.el8.elrepo.x86_64
 Operating System: Alpine Linux v3.22 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 125.8GiB
 Name: eae77d15590e
 ID: 024926f0-57c7-4049-b2a5-c3e33d4dca88
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false

Additional Info

No response

[julien-sysadmin-bzh]

Same problem here, we had to downgrade to previous docker version.

[carlos-vl]

I also had to downgrade to a previous version for this exact reason

[gdegoulet]

Nothing should have precedence when you run explictly "docker login -u" !

[julien-sysadmin-bzh]

Problem reproduced with following gitlab-ci.yml example:

stages:
  - build_image

variables:
  DOCKER_AUTH_CONFIG:
    value: "{\"auths\": {\"myrepo.mydomain.com\":{\"auth\": \"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"}}}"
    description: "docker auth"

build_test:
  extends:
    - .container-image:base
  stage: build_image
  variables:
    CONTAINER_IMAGE_NAME: test
    CONTAINER_IMAGE_TAG: 1.0.0

.container-image:base:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $CI_REGISTRY/$CI_PROJECT_PATH/$CONTAINER_IMAGE_NAME:$CONTAINER_IMAGE_TAG .
    - docker push $CI_REGISTRY/$CI_PROJECT_PATH/$CONTAINER_IMAGE_NAME:$CONTAINER_IMAGE_TAG

• DOCKER_AUTH_CONFIG is defined as global variable
• docker login is performed and should take precedence over the variable according to previous comment

case 1: DOCKER_AUTH_CONFIG is a valid token but not for current repository

docker push is refused with the following message

denied: requested access to the resource is denied

case 2: DOCKER_AUTH_CONFIG is not a valid token

docker push is performed with credentials from docker login

Failed to create credential store from DOCKER_AUTH_CONFIG:  illegal base64 data at input byte 44

[tobiasbaehr]

My workaround unset DOCKER_AUTH_CONFIG where I use docker login.

[fabriceclementz]

My workaround unset DOCKER_AUTH_CONFIG where I use docker login.

Thanks for the suggestion — I'm actually using the same workaround (unset DOCKER_AUTH_CONFIG) before docker login, and it does solve the issue for now.

However, from my point of view, this workaround isn't ideal or sustainable in the long run, as it requires modifying all affected pipelines. The current behavior also feels a bit unintuitive: without any warning or log indicating that DOCKER_AUTH_CONFIG takes precedence over docker login, it can lead to a lot of wasted debugging time for developers trying to understand why their push fails despite having valid credentials.

It would be helpful if either the precedence was more clearly documented or if the CLI emitted a warning when credentials from docker login are being ignored in favor of DOCKER_AUTH_CONFIG.

If needed and acceptable, I would be happy to help contribute a patch — whether to improve the behavior or to make the precedence and fallback logic more transparent through clearer messaging or documentation.

[carlos-vl]

Yes, at least a warning on the command line output would be helpful. My pipelines started failing suddenly and I had to spend some time trying ot understand the cause. Such a warning would have made it obvious and saved me some time.

[Benehiko]

Hi all, apologies for the disruption this has caused you 😔.

We tried to align the variable name with what is currently in use in the ecosystem and took inspiration from GitLab documentation as well as Testcontainers. Unfortunately I did not predict that CI workflows would be affected if they also relied upon docker login after setting DOCKER_AUTH_CONFIG - which was an oversight on my part.

The intent was to natively support DOCKER_AUTH_CONFIG in the CLI so that CI workflows wouldn't need to write to the Docker config file (./docker/config.json) or do a docker login. In most cases a docker login to a registry only does a credential check and does not issue a token. That means setting DOCKER_AUTH_CONFIG and doing a docker login would be the same. The DOCKER_AUTH_CONFIG environment variable takes precedence over the config file to keep the behavior consistent with how other software on Unix function (env > config).

That said, we are looking into a fix so that we can still move forward with supporting DOCKER_AUTH_CONFIG while preserving backwards compatibility with docker login usage.

[wdoekes]

Thanks for acknowledging the disruption 👍

My 2 cents: don't change behaviour now. We've found it and have done the appropriate workarounds. (And will happily use the new feature, which is nice.)

As @carlos-vl mentioned: a welcome addition would be a warning on docker login ... alerting people of the new/changed behaviour.

[heresie]

Hi @Benehiko, thank you for your response.

I respectfully disagree with @wdoekes regarding the urgency of the patch release.

The current workaround - temporarily updating all CI/CD DOCKER_AUTH_CONFIG credentials to use read/write accounts on registries - should be avoided as much as possible. It violates the principle of least privilege, which we consider essential to uphold.

Furthermore, modifying all CI/CD pipelines across multiple projects and GitLab instances to include an unset DOCKER_AUTH_CONFIG command is not a viable solution on our side. It would impact too many teams and repositories, making the change both costly and error-prone.

More broadly, this approach contradicts the widely accepted configuration precedence model used in most open-source tools. Typically, configuration sources are prioritized as follows (from lowest to highest):

1. Built-in defaults
2. Global configuration files (e.g., /etc/, ~/.config/)
3. Environment variables
4. User or project-specific configuration files
5. Explicit commands (e.g., docker login)

In this model, explicit user commands like docker login are expected to take precedence over all other sources, including environment variables such as DOCKER_AUTH_CONFIG. Ignoring that priority breaks consistency and undermines user expectations.

As @gdegoulet rightly pointed out, this behavior should remain authoritative and predictable.

[wdoekes]

@heresie

modifying all CI/CD pipelines across multiple projects and GitLab instances to include an unset DOCKER_AUTH_CONFIG command is not a viable solution on our side. It would impact too many teams and repositories, making the change both costly and error-prone.

That’s a fair point — I can’t reasonably estimate how much work this would be for others.

@gdegoulet

Nothing should have precedence when you run explicitly "docker login -u" !

Here, I respectfully disagree. In many Unix/Linux tools and workflows, environment variables are conventionally considered to have higher precedence than configuration files — precisely to enable runtime overrides without persistent changes.

I believe the confusion comes from differing interpretations of what docker login is supposed to represent:

• If we treat it as starting a session, then yes — the credentials provided should "win".
• But if it merely modifies ~/.docker/config.json, and subsequent commands might use different environment context, then environment variables should take precedence.

In the Unix philosophy, commands are typically stateless and composable. We're not starting sessions — we're executing independent processes. So from that perspective, it makes sense to me that an environment variable should override what's in the config file.

[gdegoulet]

@wdoekes I understand, it's not trivial... running "docker login -u" means we want to use those credentials for the rest of the session.

Example: If DOCKER_HOST_CONFIG is defined and contains the credentials for "REGISTRY A" and we run docker login -u xxx -p xxx "REGISTRY A" in a pipeline, this should disable (unset) DOCKER_HOST_CONFIG for the rest of the session (but only if it's the same REGISTRY), because we want to use the new credentials from "docker login".

An alternative implementation could be to override DOCKER_AUTH_CONFIG if "docker login" is for the same registry.

[wdoekes]

we want to use those credentials for the rest of the session

I hate to break it to you, but there is no session. I emplore you to take a moment to read up on how running commands work (fork/execve), and how (exported) environment variables are handled between parent/child processes.

(Resorting to keeping track of parent pids and using those, could work, but would create new problems with unpredictable behaviour: stale pids; how to handle grandchildren; ...)

An alternative implementation could be to override DOCKER_AUTH_CONFIG if "docker login" is for the same registry.

Certainly, an option in theory, but that's not how the environment works. You cannot have a child process (docker login) set/unset variables in the parent (bash) without resorting to evil hacks.

[gdegoulet]

@wdoekes : "session" is not the best word to describe the situation: you can understand it as "pipeline" ou "script" (a list of command in the same environnement )

i know that a child processes can't modify the parent environment ...it should be implemented inside docker "parts"

case 1

• DOCKER_AUTH_CONFIG is set (for "registry A")
• .docker/config.json exists and contains creds for "registry A" => If i understand, you want to use DOCKER_AUTH_CONFIG to override config file without changing file.

case 2

• DOCKER_AUTH_CONFIG is set (for "registry A")
• "docker login" is run to add creds for "registry A" => .docker/config.json is created/modified : we are now back to the previous case but we want to use new creds. how to tell next "docker commands" to not use DOCKER_AUTH_CONFIG for "registry A" ?

[bryfar-123]

Hello, I completely share @heresie @fabriceclementz @gdegoulet opinions : this change is catastrophic.

In our company, the DOCKER_AUTH_CONFIG variable is used to allow our Gitlab runners to upload technical images that are shared by all teams, into specific repositories of our private registry. This variable is defined in the GITLAB configuration, so each runner inherits it. Then, in their gitlab jobs, the developers make a docker login using their own credentials to access their own image repositories, in the same registry.

Asking them to disable DOCKER_AUTH_CONFIG in their jobs is not an acceptable solution for us.

[wdoekes]

Dear @bryfar-123,

I won't try to downplay the impact is has. I'm sorry to hear it is catastrofic for your current situation.

Asking them to disable DOCKER_AUTH_CONFIG in their jobs is not an acceptable solution for us.

But, ultimately the problem is how the GitLab runners are interacting with the variable, not necessarily a problem with Docker cli.

https://gitlab.com/gitlab-org/gitlab-runner/-/blob/549b102ce4249d7cc2266f283a24e83b57a50c30/helpers/docker/auth/auth.go#L27 (here the GitLab runner chooses the variable to use)

https://gitlab.com/gitlab-org/gitlab-runner/-/blob/549b102ce4249d7cc2266f283a24e83b57a50c30/helpers/docker/official_docker_client.go#L319 (no environment variables are passed to the docker libs, only actual secrets)

So yes, adding DOCKER_AUTH_CONFIG support to docker-cli breaks the current setup for some people now. But arguing that docker-cli must be "fixed" is not necessarily true. Maybe the GitLab-runner needs to be fixed/patched instead. If they had called it GITLAB_RUNNER_DOCKER_AUTH in the first place we wouldn't be in this mess.

Don't forget that the docker login in gitlab CI scripts has been a permanent eyesore. I can only say I approve that there finally is a logical/proper way to set the auth.

(Yes, I'm thinking long term here. Anything that mitigates things in the mean time is certainly fine. But don't undo the good just because there is extra work in the short term. Sometimes you have to bite the bullet to get improved behaviour/sanity down the line.)

[edit: Maybe file a bug at GitLab. With their track record, they might get around to it in 2035. /s]

[el-liberator]

But why was this change, which may seem minor for some but very impacting for many people, not announced beforehand ? , the simple: "Use DOCKER_AUTH_CONFIG as a credential store" in version 28.3.0-rc.2 is not enough

[wdoekes]

But why was this change, which may seem minor for some but very impacting for many people, not announced beforehand ? , the simple: "Use DOCKER_AUTH_CONFIG as a credential store" in version 28.3.0-rc.2 is not enough

Obviously because the developers did not anticipate the fallout. And I don't think you can blame them.

• GitLab first uses its own runner infra and the DOCKER_AUTH_CONFIG variable to pull runner images from docker (docker go libs, not docker cli). (Questionable naming by GitLab; I don't go around naming my variables GOOGLE_* and then expect the google-team to not reuse those for something else.)
• GitLab users then use docker daemon and cli (docker-in-docker) to build stuff. To push (and maybe pull) they auth locally (and sometimes with different auth -- which is when the problems surface). (Questionable usage by users. Ideally there would be a nice way to interface with Docker through the runner instead of separately / using the cli.)
• After the change, an unused environment variable is suddenly used. (It was used by the runner, but not by the runner scripts. But because of how things work it gets exported to the scripts as well. And in the scripts it's suddenly a problem.)

I'm not using the word "Questionable" to mean that they were Wrong. They just wanted to get stuff to work. But this interaction is not "logical" or "expected" from the perspective of someone working on Docker and Docker-cli.

[vvoland]

But why was this change, which may seem minor for some but very impacting for many people, not announced beforehand ? , the simple: "Use DOCKER_AUTH_CONFIG as a credential store" in version 28.3.0-rc.2 is not enough

Hi, I'm sorry the change caused unexpected behavior on your side!

I'm happy to hear how do you think we could improve the process to communicate such changes in the future. What communication channels would make it easier for you to discover new RCs?

[el-liberator]

But why was this change, which may seem minor for some but very impacting for many people, not announced beforehand ? , the simple: "Use DOCKER_AUTH_CONFIG as a credential store" in version 28.3.0-rc.2 is not enough

Hi, I'm sorry the change caused unexpected behavior on your side!

I'm happy to hear how do you think we could improve the process to communicate such changes in the future. What communication channels would make it easier for you to discover new RCs?

Hi, Maybe by creating a new page like the deprecated page (https://github.com/docker/cli/blob/v28.3.0/docs/deprecated.md), explaining the novelties and their impacts

[heresie]

@wdoekes

Thank you again for the detailed explanation. I understand and respect the goal of creating a more consistent and secure mechanism for handling Docker authentication. Using DOCKER_AUTH_CONFIG as a credential store makes sense in many scenarios and is a logical step forward.

That said, I believe the way this change was introduced deserves reconsideration. The impact was significant for many CI/CD workflows, especially where DOCKER_AUTH_CONFIG was previously ignored by the Docker CLI. In these environments, inherited variables can unintentionally override expected authentication flows. A changelog line like "Use DOCKER_AUTH_CONFIG as a credential store" was not enough to communicate the potential impact.

There are two areas where I think improvements would make a meaningful difference:

1. Better visibility and feedback: We saw the PR adding verbosity (https://github.com/docker/cli/pull/6163), which is a very positive step. It would help even more if the CLI could provide clearer warnings when DOCKER_AUTH_CONFIG is active and potentially interfering with commands like docker login. That visibility would help users debug and adapt their pipelines much faster.

2. Support for stateless CI/CD environments: GitLab CI/CD pipelines, like many others, are intentionally stateless. Each job runs inside a clean container with no persisted context or user configuration between steps. In this model, users often rely on scripts to perform authentication dynamically, including running docker login at runtime. If DOCKER_AUTH_CONFIG is set by the CI system or inherited from the runner, it can unintentionally override these steps.

   What we need is not for Docker to ignore DOCKER_AUTH_CONFIG altogether, but for it to offer a safe and flexible way to override or bypass it when necessary. For example, a daemon.json option or a new environment variable like DOCKER_IGNORE_AUTH_CONFIG could explicitly disable use of the variable during docker login. This would allow CI workflows to maintain precise control over credentials, without having to unset or sanitize environment variables manually.

At the same time, this ties into broader security practices. In CI environments, we encourage least-privilege principles and tightly scoped credentials. To support that model, Docker should make it easier, not harder, for users to switch or override credentials cleanly at runtime, whether through environment variables or direct commands.

I agree that GitLab might want to revisit its use of the variable, but once a name like DOCKER_AUTH_CONFIG is used across multiple tools and workflows, it becomes critical to roll out behavior changes carefully. Docker is in the best position to lead that transition and provide the flexibility needed to support different environments safely.

This is not about assigning blame. It is about improving resilience and user experience, especially in production-grade, automated, and stateless setups. I hope the Docker team will continue to iterate on both the communication and the flexibility around this feature.

[vvoland]

2. What we need is not for Docker to ignore DOCKER_AUTH_CONFIG altogether, but for it to offer a safe and flexible way to override or bypass it when necessary. For example, a daemon.json option or a new environment variable like DOCKER_IGNORE_AUTH_CONFIG

If you don't want Docker to use DOCKER_AUTH_CONFIG, why not unset it/set it to empty?

[vvoland]

As a temporary hot-fix, we're considering disabling that feature by default when running in Gitlab runner.

I'm thinking about the best way to detect that.. Looking at https://docs.gitlab.com/ci/variables/predefined_variables/ it looks like we could check for GITLAB_CI environment-variable.

Could you verify if that variable would indeed be set in your case?

[heresie]

@vvoland

That’s a fair question, and unsetting DOCKER_AUTH_CONFIG can work in many simple cases. But in practice, especially in CI/CD environments like GitLab, there are a few reasons why relying solely on unsetting the variable is not always sufficient or practical:

1. Limited control over the environment: In GitLab CI/CD, the environment is often prepared by the runner infrastructure. Variables like DOCKER_AUTH_CONFIG may be injected automatically, for example through GitLab’s own credential helper or Docker-in-Docker configuration. Users running jobs do not always have visibility into or control over when and how this variable is set.

2. Stateless and isolated job execution: Each CI job runs in a clean, isolated container. There is no persistent user configuration between jobs. As a result, every job script would need to detect and unset DOCKER_AUTH_CONFIG manually, which increases complexity and introduces inconsistency across pipelines.

3. Need for flexibility and separation of credentials: In CI workflows that follow least-privilege principles, different pipeline stages may use different sets of credentials. For example, one stage may pull images using credentials defined via DOCKER_AUTH_CONFIG, while another stage needs to push using credentials set via docker login. In such cases, we do not want to eliminate DOCKER_AUTH_CONFIG entirely, only to make sure it does not interfere with specific commands like docker login.

4. Predictability and long-term maintainability: If the Docker CLI offered a clear, documented way to ignore or override DOCKER_AUTH_CONFIG, such as a setting in daemon.json or an opt-out variable like DOCKER_IGNORE_AUTH_CONFIG, users could more easily manage their workflows without relying on shell hacks or environment sanitization.

It is worth noting that the decision to use the DOCKER_AUTH_CONFIG name was intentional. As @Benehiko pointed out, the Docker team tried to align with naming already in use across the ecosystem, including references in GitLab documentation and tools like Testcontainers. However, this also means the team was aware that the variable was already used in CI environments, which makes it even more important to provide safeguards when changing how the CLI interprets it. The oversight was not in choosing the name, but in underestimating the side effects on CI workflows where docker login is used after the variable is set.

To be clear, this is not a request to remove support for DOCKER_AUTH_CONFIG. It is a request to give users more explicit control. When environment behavior affects security and credentials, it is critical to provide ways to make that behavior predictable and auditable - especially in stateless CI systems where clarity and control are essential.

[gdegoulet]

To be honest, I thought DOCKER_AUTH_CONFIG was a DOCKER configuration variable and not something related to GITLAB's Docker executor! @wdoekes is right to question the validity of naming the variable like that !

In any case, the problem still remains:

• It seems difficult to ask GITLAB to change the variable name: it would break even more things.
• Ask GITLAB to modify/limit the scope of the variable to only allow the "initial pull" for the "base" image and image services (docker executor).
• Ask Docker to revert this change? Stop offering an environment variable and instead suggest users create a temporary configuration file on the fly and use it?
• Ask Docker to change the variable name to reflect existing usages? Something like DOCKER_AUTH_ENV instead of DOCKER_AUTH_CONFIG? In this case, using the "docker login" command should generate an error or warning if DOCKER_AUTH_ENV is present and refers to the same registry (you can perfectly well want to connect to another registry while DOCKER_AUTH_ENV exists).
• Last solution: make users assume this new docker feature: DOCKER_AUTH_CONFIG is now a "docker" variable (also): modify your pipelines accordingly (unset DOCKER_AUTH_CONFIG at the beginning of the pipeline).

[heresie]

@vvoland

As a temporary hot-fix, we're considering disabling that feature by default when running in Gitlab runner.

I'm thinking about the best way to detect that.. Looking at https://docs.gitlab.com/ci/variables/predefined_variables/ it looks like we could check for GITLAB_CI environment-variable.

Could you verify if that variable would indeed be set in your case?

Yes, the GITLAB_CI environment variable is always set to true when a job runs in a GitLab CI/CD pipeline. This is a predefined GitLab environment variable and can reliably be used to detect when the code is running in a GitLab runner.

[bryfar-123]

as @heresie says , Docker team told they wanted to align on this DOCKER_AUTH_CONFIG variable used in Gitlab, for better integration. So asking Gitlab to change the name of this variable, or disabling that feature by default when running in Gitlab runner would solve our problems, but that would be a nonsense, wouldn’t it?

[wdoekes]

@vvoland:

As a temporary hot-fix, we're considering disabling that feature by default when running in Gitlab runner.

Do you maybe mean: ".. considering reordering the precedence of file-auth vs. env-auth [...] when running in Gitlab runner" ?

Because if you're disabling the env auth in the runner scripts, we're back to the dark ages where we're forced to abuse docker login.

[gdegoulet]

echo $DOCKER_AUTH_CONFIG | jq .
{
  "auths": {
    "my.docker-registry.com": {
      "auth": "cm9ib3Qkc3lzdGVtZS1wdWJsaWMtd3JpdGU6WFhYWFhYWFhYWFhYWFhYWFhYWFhY"
    }
  },
  "docker_cli_auto_login": true
}

one solution could be to add a new key/value in the json structure : something like above. if "docker_cli_auto_login" is not set or false, docker may ignore DOCKER_AUTH_CONFIG

in this case, there is no breaking change in gitlab pipelines and the new feature is possible ...

[vvoland]

@wdoekes the initial intent was a quick fix before we get: https://github.com/moby/moby/pull/50341 (which would basically allow trying the next available credentials instead of just failing fast on the first attempt).

Adding an option to configure the order of preference would also work though, and probably also align nicely with the mentioned PR.