Allow env_files in docker-compose files to be optional

[deviantintegral]

Description

As a development leader, I would like teams to be able to spin up local docker-compose based environments without steps on the host beyond git clone.... As well, developers need to be able to optionally set or override environment variables such as connection strings or secrets. Since overriding variables is the 20% case, most developers should be able to run:

git clone <url>
docker-compose up -d

One of the most common patterns is to have a checked-in .env file of defaults, with .env.local or similar overriding the first file. Ideally, this could be achieved with:

service:
  web:
    env_file:
      - .env
      - .env.local

However, docker-compose requires .env.local to exist, and errors out. This is especially problematic with VS Code's "clone in named volume" functionality (key for working around Docker performance issues on macOS and Windows) in the remote containers extension, as there's no opportunity to run a script to create even a stub file before containers are started.

Additional information you deem important (e.g. issue happens only occasionally):

There was a previous declined PR at https://github.com/docker/compose/pull/3955 from 2017, along with a related issue at https://github.com/docker/compose/issues/3560. However, the PR has many comments since closing detailing common use cases and hacks around this missing functionality, as well as comparison to existing tools that do treat .env files as optional. I'm opening this issue for current feedback from current maintainers, as it seems that the original closer no longer works at Docker. I've likely missed suggestions and edge cases around this feature expressed in the PR comments, so please don't take the above as a complete description of community needs.

Open questions

• Is there planning underway or other issues which would actually solve this use case, but in a non-obvious way? I ask because there was a similar set of issues around making individual services optional that were eventually solved by profiles. I know that wasn't obvious to me until the profiles feature actually landed, so I have some hope there's something in compose v2 to address this.
• Otherwise, can we get an update from maintainers on current thinking around this issue? Even if that's to close this as "won't fix", given 5 years has passed that would still be a valuable signal that users should continue to find workarounds or alternate tools / methodologies to support this workflow.

[ndeloof]

Seems to me there's a confusion here between .env file used to pass variables to compose, and env_file used to set container environment, which is just the exact same as docker run --env-file. Making the later optional would trigger a bunch of new issues and confusion.

The recommended way to support "optionally set or override environment variables" is to declare those as service environment without a value, so that they get interpolated from environment variables and/or local .env file

service:
  web:
     - SOME_VAR

if SOME_VAR is not declared, container will be created without such a variable set. Otherwise current value will be set.

[ndeloof]

Assuming we want to add support for "optional env_file" as you describe, this should be debated under Compose specification, and probably will better be addressed with a custom syntax to mark file as optional, maybe using elvis operator ?:

service:
  web:
    env_file:
      - .env
      - ?:.env.local # optional file

[Kallys]

I'm interessed by this feature for same motivations (working docker compose up just after a git clone) as it may resolve my concerns about default env values.

Illustration

Let's look at this code for example (inspired from this one).

# docker-compose.yml
services:
  php:
    environment:
      DATABASE_URL: postgresql://${POSTGRES_USER:-api-platform}:${POSTGRES_PASSWORD:-!ChangeMe!}@database:5432/${POSTGRES_DB:-api}?serverVersion=${POSTGRES_VERSION:-13}

  database:
    image: postgres:${POSTGRES_VERSION:-13}-alpine
    environment:
      - POSTGRES_DB=${POSTGRES_DB:-api}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-!ChangeMe!}
      - POSTGRES_USER=${POSTGRES_USER:-api-platform}

If someone modify POSTGRES_DB default value in "database" service forgetting doing the same in "php" service, it could lead to some difficulties...

Since there is no ability to define global constants in compose files and yaml anchors variables are not expanded, I tried using .env file as a way to both expose and provide environment default values.

# docker-compose.yml
services:
  php:
    environment:
      DATABASE_URL: postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@database:5432/${POSTGRES_DB}?serverVersion=${POSTGRES_VERSION}

  database:
    image: postgres:${POSTGRES_VERSION}-alpine
    environment:
      - POSTGRES_DB=${POSTGRES_DB}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_USER=${POSTGRES_USER}

# .env
POSTGRES_DB=api-platform
POSTGRES_PASSWORD=!ChangeMe!
POSTGRES_VERSION=13
POSTGRES_USER=api-platform

This solution works great ! But I came out to the same issue when I tried to commit this .env file and wanted to allow team members to locally override those values...

[deviantintegral]

Thanks @ndeloof , this is helpful. The above example really summarizes this issue with:

But I came out to the same issue when I tried to commit this .env file and wanted to allow team members to locally override those values...

Also:

declare those as service environment without a value

So it seems like this is somewhat doable as long as defaults aren't provided. That seems to break the 80% case of working "out of the clone" without additional configuration.

I think your proposal with the ? operator would work nicely.

[q0rban]

What about allowing glob patterns such as .env(.local|)

service:
  web:
    env_file:
      - '.env(.local|)'

[ndeloof]

Please, do not try to include some pseudo-programing langage / expression in compose file format :P

[ghost]

Hi I really want to have this option.

env_file:
  - path: ''./web/.env.development.local'
    required: false
  - path: ''./web/.env.local'
    required: true

How about this format. The advantage of this is capable of many env_files while some is not required and others are required. It happens many times when you have Unit Test CI while you also have E2E Test CI.

If maintainer agrees with it I could create PR.

[ndeloof]

This should first be debated on compose-specification

[ghost]

@ndeloof

yes, i saw you said that in previous comment as well, but no issue is submitted in compose-specification yet. so i wondered if there is still a problem.

you said 'this should be debated in compose-specification' but does it mean 'please post a issue in compose-specification' or 'please wait' ?

i'm just confused.

[ndeloof]

@ulwlu compose spec is public, anyone can open a proposal. If you feel this is un important topic, you're welcome to do

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[quirin-buechner-mdctec]

What about allowing glob patterns such as *.env using a syntax we all know well from .dockerignore files

service:
  web:
    env_file:
      - '*.env'

this would match both files .env and .local.env

[diddi-]

What about allowing glob patterns such as *.env using a syntax we all know well from .dockerignore files

service:
  web:
    env_file:
      - '*.env'

this would match both files .env and .local.env

This would allow for multiple env files sure, but it doesn't help with the order of precedence so the only way to override the "global" .env is to make sure the local env file is alphabetically sorted first.

[landsman]

yes please!

[vikyw89]

please !

[hopeng]

please!

[retog]

This issue causes problem using vscode devcontainers and codespaces. Without the .env file the container can't be built. When I add an .env to git there's (afaik) no way to tell git to ignore future changes to that file, so the developer has to manually make sure not to accidentally commit secrets and the codespace will be listed as having uncommitted changes.

[AlexSkrypnyk]

The Compose Spec has accepted this change in https://github.com/compose-spec/compose-spec/issues/240

I suggest refrain from discussing changes to the format/syntax or the reasons we need this (we all have them!) as it is now a matter of the spec implementation in this repo.

[bybatkhuu]

Is it working? Mine doesn't work:

compose.yml:

  api:
    image: somerepo/somename:sometag
    depends_on:
      - db
    restart: unless-stopped
    env_file:
      - path: ./.env
        required: false

Output error:

validating /Users/someuser/workspaces/projects/test/compose.yml: services.api.env_file.0 must be a string

Docker version:

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:04:20 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.26.1 (131620)
 Engine:
  Version:          master
  API version:      1.44 (minimum version 1.12)
  Go version:       go1.21.3
  Git commit:       54fcd40aa4de94cd75aedc5f6ebf38c6d8f92082
  Built:            Wed Nov 22 07:43:39 2023
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker compose version:

Docker Compose version v2.23.3-desktop.2

[ndeloof]

This is available in Docker Compose v2.24 and later