Wrong usage of env_file in docker-compose.middleware.yaml

[jter]

Self Checks

• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [X] Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

The way we use env_file in docker-compose.middleware.yaml is probably a mistake.

Variables in middleware.env are used for docker compose itself rather than setting them inside the container

As per description of env_file in docker compose's official document, env_file is used for setting environment variable in
container

The correct ways to use middleware.env are:

1. Rename it to .env, it will be loaded by docker compose when we use docker compose up
2. Specify explicitly --env-file parameter in docker compose --env-file ./middleware.env up (this is totally equal different thing from env_file in docker-compose.middleware.yaml)

Solution for this issue is simple: Just remove all env_file in docker-compose.middleware.yaml (No problem found in docker-compose.yaml)

doc of docker compose: https://docs.docker.com/compose/how-tos/environment-variables/

Please check it out, if this issue can be confirmed, I can make a PR.

2. Additional context or comments

No response

3. Can you help us with this feature?

• [X] I am interested in contributing to this feature.

[dosubot[bot]]

Hello @jter! I'm here to assist you with any bugs, questions, or contributions you may have. Let me know how I can help you today.

The current usage of env_file in docker-compose.middleware.yaml is indeed to load environment variables from the middleware.env file into the db and weaviate services ^[1]. This approach sets the environment variables inside the containers, which aligns with the intended use of env_file in Docker Compose.

However, if the intention is to use these variables for Docker Compose itself rather than inside the containers, you might consider the following alternatives:

1. Rename middleware.env to .env: This allows Docker Compose to automatically load the environment variables when you run docker compose up, applying them to the entire Compose process.

2. Use the --env-file parameter: You can explicitly specify the environment file with docker compose --env-file ./middleware.env up, which is a different mechanism from using env_file within the Compose file.

If the goal is to apply these variables globally to the Docker Compose process rather than to specific services, these alternatives could be more appropriate. Removing env_file from docker-compose.middleware.yaml would be a valid solution if the variables are not needed inside the containers.

To continue talking to Dosu, mention @dosu.

[Hisir0909]

docker compose --env-file ./middleware.env up and specifying env-file in the compose file are only different in priority.

The order of precedence (highest to lowest) is as follows:

1. Set using docker compose run -e in the CLI.
2. Set with either the environment or env_file attribute but with the value interpolated from your shell or an environment file. (either your default .env file, or with the --env-file argument in the CLI).
3. Set using just the environment attribute in the Compose file.
4. Use of the env_file attribute in the Compose file.
5. Set in a container image in the ENV directive. Having any ARG or ENV setting in a Dockerfile evaluates only if there is no Docker Compose entry for environment, env_file or run --env.

[jter]

docker compose --env-file ./middleware.env up and specifying env-file in the compose file are only different in priority.

The order of precedence (highest to lowest) is as follows:

1. Set using docker compose run -e in the CLI.
2. Set with either the environment or env_file attribute but with the value interpolated from your shell or an environment file. (either your default .env file, or with the --env-file argument in the CLI).
3. Set using just the environment attribute in the Compose file.
4. Use of the env_file attribute in the Compose file.
5. Set in a container image in the ENV directive. Having any ARG or ENV setting in a Dockerfile evaluates only if there is no Docker Compose entry for environment, env_file or run --env.

Whaaaat? issue closed? @crazywoola

Yes, you are correct, I misunderstood env_file attribute

But let me clarify this

First, as doc describes, if you want to use Interpolation function, you'll have to use --env-file in the command line of docker compose --env_file=xxx.env or variables in xxx.env won't be substituted in docker-compose file. here's an testing example:

TEST1:

file: test.env
VALUE=test
TEST_VAL=FANCY_NAME

file: docker-compose.yml
services:
  web:
    image: "bash"
    env_file: ./test.env
    environment:
      - TEST_ENV=${VALUE}
      - ${TEST_VAL}=1
     tty: true

run command: docker compose up output:

➜test-compose (ml) docker compose up
WARN[0000] The "VALUE" variable is not set. Defaulting to a blank string.
WARN[0000] The "TEST_VAL" variable is not set. Defaulting to a blank string.
[+] Running 4/4
 ✔ web Pulled                                                                                                                                                                7.6s
   ✔ 43c4264eed91 Pull complete                                                                                                                                              3.3s
   ✔ 467989b961c3 Pull complete                                                                                                                                              3.6s
   ✔ e1517fbc9c07 Pull complete                                                                                                                                              3.6s
[+] Running 0/1
 ⠋ Container test-compose-web-1  Recreate                                                                                                                                    0.1s
**Error response from daemon: invalid environment variable: =1**

Error happened, because Interpolation hadn't been done, so ${TEST_VAL} was set to blank string

TEST2: Keep all file contents the same as TEST1 run command: docker compose --env-file=test.env up OUTPUT:

➜  test-compose (ml) docker compose --env-file=test.env up
[+] Running 2/2
 ✔ Network test-compose_default  Created                                                                                                                                     0.1s
 ✔ Container test-compose-web-1  Created                                                                                                                                    0.1s

container run successfully

run command: docker exec -it test-compose-web-1 bash

bash-5.2# printenv

FANCY_NAME=1
TEST_VAL=FANCY_NAME
HOME=/root
_BASH_VERSION=5.2.37
_BASH_BASELINE=5.2.37
_BASH_LATEST_PATCH=37
TERM=xterm
TEST_ENV=test
VALUE=test
SHLVL=1

FANCY_NAME=1 indicates Interpolation happened.

So this question transforms as What's the purpose of middleware.env? To let people configure things which needs to be done during Interpolation or just configure envs inside containers?

Currently most variables in middleware.env have different names with variables defined in environment attribute

Make a decision

[Hisir0909]

@jter I understand what you mean. Using both env_file and environment in compose.yaml, and the .env file not existing, results in the variables in environment being empty, correct? This causes the middleware.env configuration to be invalid, right?

[jter]

@jter I understand what you mean. Using both env_file and environment in compose.yaml, and the .env file not existing, results in the variables in environment being empty, correct? This causes the middleware.env configuration to be invalid, right?

Yes

Currently, configuration changes in middleware.env won't take effect, unless specific --env-file in command line. env-file in docker-compose.middleware.yaml does nothing, what actually take effect are those default value in docker-compose.middleware.yaml.

So, are we going to reopen this issue and make things right? @crazywoola