dagu icon indicating copy to clipboard operation
dagu copied to clipboard
verified publisher icon dagu-dev

bug: inconsistent environment variables behaviour

Open thibmart1 opened this issue 2 months ago • 13 comments

Hello

I tried to use Dagu for the past couple days (awesome work by the way, thank you so much!) and noticed some inconsistencies with environment variables handling between the different ways to execute dags.

First, the environment variables reference does not work for me with manual execution from WebUI or from the scheduler.

The following :

tags: [backup, docker]
schedule: "0 1 * * *"
env:
  DIR_DOCKER: ${DIR_DOCKER}
steps:
  - name: test 1
    command: id
  - name: test 2
    command: env
  - name: cleanup old backups
    command: find "${DIR_DOCKER}/data/backup" -type f -mtime +5 -exec rm {} \;

fail on 3rd step because ${DIR_DOCKER} is not resolved, which seems to be a bug in this mentioned featured in the doc.

I also tried to run the dags directly from CLI (dagu start myDag.yaml) and it worked but because the all env from my user was passed to Dagu instead of being filtered (as I understand it).

In all three ways of running the dag (manually from the WebUI, with the scheduler and from the CLI), the outputs of the first command were the same (ie. the same user was used to run the commands set).

Thank you in advance for your help on this one :)

thibmart1 avatar Oct 12 '25 20:10 thibmart1

One basic query: Is the scheduler/ui running from same user from which dagu start myDag.yaml command was used?

Can you try prefixing: Allowed Prefixes: DAGU_*, LC_*,DAG_*

in the DIR_DOCKER environment variable and check for webui/scheduler cases?

What is the output of step 2 in all 3 cases? That may give a clue

ghansham avatar Oct 13 '25 00:10 ghansham

Here are the different results of the first and second step for webui/scheduler/cli cases after replacing DIR_DOCKER: ${DIR_DOCKER} by DIR_DOCKER: ${DAG_DIR_DOCKER} and set the DAG_DIR_DOCKER variable in the .profile of my user.

Again, only the cll case worked.

Step 1

# scheduler
uid=1000(captain) gid=1000(captain) groups=1000(captain)

# webui
uid=1000(captain) gid=1000(captain) groups=1000(captain)

# cli
uid=1000(captain) gid=1000(captain) groups=1000(captain)

Step 2

# scheduler
5=ARCHIVE=
4=REPOSITORY=
3=ARCHIVE=
2=RETURN_CODE=
1=ENDPOINT=
SHELL=/bin/bash
LANGUAGE=en_US:en
DIR_DOCKER=
DAG_RUN_STEP_STDOUT_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_060100Z_0199dc28-6961-7985-921b-00aa7192f240/run_20251013_060100Z_7975a8/cmd_2.20251013.080100.147.0199dc28.out
DAG_RUN_STEP_NAME=cmd_2
DAG_RUN_LOG_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_060100Z_0199dc28-6961-7985-921b-00aa7192f240/dag-run_20251013.080100.028.0199dc28.log
PWD=/nautilus/apps/dagu/dags
LOGNAME=captain
DAG_RUN_ID=0199dc28-6961-7985-921b-00aa7192f240
SYSTEMD_EXEC_PID=3419255
TZ=Europe/Paris
DAG_NAME=myDag
HOME=/home/captain
LANG=en_US.UTF-8
DAG_RUN_STEP_STDERR_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_060100Z_0199dc28-6961-7985-921b-00aa7192f240/run_20251013_060100Z_7975a8/cmd_2.20251013.080100.147.0199dc28.err
INVOCATION_ID=a87b8ea529d4478e826aa9c981c1b831
USER=captain
SHLVL=0
JOURNAL_STREAM=8:380334880
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env

# webui
5=ARCHIVE=
4=REPOSITORY=
3=ARCHIVE=
2=RETURN_CODE=
1=ENDPOINT=
SHELL=/bin/bash
LANGUAGE=en_US:en
DIR_DOCKER=
DAG_RUN_STEP_STDOUT_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_061004Z_0199dc30-b78c-7ce3-9406-c8def329d88e/run_20251013_061004Z_0ef54c/cmd_2.20251013.081004.429.0199dc30.out
DAG_RUN_STEP_NAME=cmd_2
DAG_RUN_LOG_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_061004Z_0199dc30-b78c-7ce3-9406-c8def329d88e/dag-run_20251013.081004.316.0199dc30.log
PWD=/nautilus/apps/dagu/dags
LOGNAME=captain
DAG_RUN_ID=0199dc30-b78c-7ce3-9406-c8def329d88e
SYSTEMD_EXEC_PID=3419255
TZ=Europe/Paris
DAG_NAME=myDag
HOME=/home/captain
LANG=en_US.UTF-8
DAG_RUN_STEP_STDERR_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_061004Z_0199dc30-b78c-7ce3-9406-c8def329d88e/run_20251013_061004Z_0ef54c/cmd_2.20251013.081004.429.0199dc30.err
INVOCATION_ID=a87b8ea529d4478e826aa9c981c1b831
USER=captain
SHLVL=0
JOURNAL_STREAM=8:380334880
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env

# cli
5=ARCHIVE=
4=REPOSITORY=
3=ARCHIVE=
2=RETURN_CODE=
1=ENDPOINT=
SHELL=/bin/bash
LANGUAGE=en_US:en
DIR_DOCKER=/nautilus/docker_apps
DAG_RUN_STEP_STDOUT_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_061004Z_0199dc30-b78c-7ce3-9406-c8def329d88e/run_20251013_061004Z_0ef54c/cmd_2.20251013.081004.429.0199dc30.out
DAG_RUN_STEP_NAME=cmd_2
DAG_RUN_LOG_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_061004Z_0199dc30-b78c-7ce3-9406-c8def329d88e/dag-run_20251013.081004.316.0199dc30.log
PWD=/nautilus/apps/dagu/dags
LOGNAME=captain
DAG_RUN_ID=0199dc30-b78c-7ce3-9406-c8def329d88e
SYSTEMD_EXEC_PID=3419255
TZ=Europe/Paris
DAG_NAME=myDag
HOME=/home/captain
LANG=en_US.UTF-8
DAG_RUN_STEP_STDERR_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_061004Z_0199dc30-b78c-7ce3-9406-c8def329d88e/run_20251013_061004Z_0ef54c/cmd_2.20251013.081004.429.0199dc30.err
INVOCATION_ID=a87b8ea529d4478e826aa9c981c1b831
USER=captain
SHLVL=0
JOURNAL_STREAM=8:380334880
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env
DAG_DIR_DOCKER=/nautilus/docker_apps

... all the rest of my env variables defined in the .profile (redacted)

Some other checks I ran on my machine directly:

captain@nas:/nautilus/apps/dagu/dags> id
uid=1000(captain) gid=1000(captain) groups=1000(captain)

captain@nas:/nautilus/apps/dagu/dags> echo $DIR_DOCKER
/nautilus/docker_apps

captain@nas:/nautilus/apps/dagu/dags> echo $DAG_DIR_DOCKER
/nautilus/docker_apps

captain@nas:/nautilus/apps/dagu/dags> ps -elf | grep dagu
4 S captain  3419255       1  0  80   0 - 181113 do_epo Oct10 ?       00:00:00 node /usr/bin/dagu start-all
0 S captain  3419267 3419255  0  80   0 - 316798 futex_ Oct10 ?       00:02:35 /usr/lib/node_modules/dagu/dagu start-all

The definition of dagu systemd service:

[Unit]
Description=Dagu Workflow Engine
Documentation=https://docs.dagu.cloud/
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=captain
Group=captain
WorkingDirectory=/nautilus/apps/dagu

# Main process
ExecStart=/usr/bin/dagu start-all

# Graceful shutdown
ExecStop=/bin/kill -TERM $MAINPID
TimeoutStopSec=30
KillMode=mixed
KillSignal=SIGTERM

# Restart policy
Restart=always
RestartSec=10
StartLimitInterval=60
StartLimitBurst=3

# Resource limits
LimitNOFILE=65536
LimitNPROC=4096

[Install]
WantedBy=multi-user.target

thibmart1 avatar Oct 13 '25 06:10 thibmart1

Can you attempt from teminal logged as captain user, try following:

echo $DAG_DIR_DOCKER

/usr/bin/dagu start-all

And then check from web ui by starting myDag.yaml.

On Mon, Oct 13, 2025, 12:03 PM thibmart1 @.***> wrote:

thibmart1 left a comment (dagu-org/dagu#1320) https://github.com/dagu-org/dagu/issues/1320#issuecomment-3396049137

Here are the different results of the first and second step for webui/scheduler/cli cases after replacing DIR_DOCKER: ${DIR_DOCKER} by DIR_DOCKER: ${DAG_DIR_DOCKER} and set the DAG_DIR_DOCKER variable in the .profile of my user.

Again, only the cll case worked.

Step 1

scheduler

uid=1000(captain) gid=1000(captain) groups=1000(captain)

webui

uid=1000(captain) gid=1000(captain) groups=1000(captain)

cli

uid=1000(captain) gid=1000(captain) groups=1000(captain)

Step 2

scheduler

5=ARCHIVE= 4=REPOSITORY= 3=ARCHIVE= 2=RETURN_CODE= 1=ENDPOINT= SHELL=/bin/bash LANGUAGE=en_US:en DIR_DOCKER= DAG_RUN_STEP_STDOUT_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_060100Z_0199dc28-6961-7985-921b-00aa7192f240/run_20251013_060100Z_7975a8/cmd_2.20251013.080100.147.0199dc28.out DAG_RUN_STEP_NAME=cmd_2 DAG_RUN_LOG_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_060100Z_0199dc28-6961-7985-921b-00aa7192f240/dag-run_20251013.080100.028.0199dc28.log PWD=/nautilus/apps/dagu/dags LOGNAME=captain DAG_RUN_ID=0199dc28-6961-7985-921b-00aa7192f240 SYSTEMD_EXEC_PID=3419255 TZ=Europe/Paris DAG_NAME=myDag HOME=/home/captain LANG=en_US.UTF-8 DAG_RUN_STEP_STDERR_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_060100Z_0199dc28-6961-7985-921b-00aa7192f240/run_20251013_060100Z_7975a8/cmd_2.20251013.080100.147.0199dc28.err INVOCATION_ID=a87b8ea529d4478e826aa9c981c1b831 USER=captain SHLVL=0 JOURNAL_STREAM=8:380334880 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin _=/usr/bin/env

webui

5=ARCHIVE= 4=REPOSITORY= 3=ARCHIVE= 2=RETURN_CODE= 1=ENDPOINT= SHELL=/bin/bash LANGUAGE=en_US:en DIR_DOCKER= DAG_RUN_STEP_STDOUT_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_061004Z_0199dc30-b78c-7ce3-9406-c8def329d88e/run_20251013_061004Z_0ef54c/cmd_2.20251013.081004.429.0199dc30.out DAG_RUN_STEP_NAME=cmd_2 DAG_RUN_LOG_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_061004Z_0199dc30-b78c-7ce3-9406-c8def329d88e/dag-run_20251013.081004.316.0199dc30.log PWD=/nautilus/apps/dagu/dags LOGNAME=captain DAG_RUN_ID=0199dc30-b78c-7ce3-9406-c8def329d88e SYSTEMD_EXEC_PID=3419255 TZ=Europe/Paris DAG_NAME=myDag HOME=/home/captain LANG=en_US.UTF-8 DAG_RUN_STEP_STDERR_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_061004Z_0199dc30-b78c-7ce3-9406-c8def329d88e/run_20251013_061004Z_0ef54c/cmd_2.20251013.081004.429.0199dc30.err INVOCATION_ID=a87b8ea529d4478e826aa9c981c1b831 USER=captain SHLVL=0 JOURNAL_STREAM=8:380334880 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin _=/usr/bin/env

cli

5=ARCHIVE= 4=REPOSITORY= 3=ARCHIVE= 2=RETURN_CODE= 1=ENDPOINT= SHELL=/bin/bash LANGUAGE=en_US:en DIR_DOCKER=/nautilus/docker_apps DAG_RUN_STEP_STDOUT_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_061004Z_0199dc30-b78c-7ce3-9406-c8def329d88e/run_20251013_061004Z_0ef54c/cmd_2.20251013.081004.429.0199dc30.out DAG_RUN_STEP_NAME=cmd_2 DAG_RUN_LOG_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_061004Z_0199dc30-b78c-7ce3-9406-c8def329d88e/dag-run_20251013.081004.316.0199dc30.log PWD=/nautilus/apps/dagu/dags LOGNAME=captain DAG_RUN_ID=0199dc30-b78c-7ce3-9406-c8def329d88e SYSTEMD_EXEC_PID=3419255 TZ=Europe/Paris DAG_NAME=myDag HOME=/home/captain LANG=en_US.UTF-8 DAG_RUN_STEP_STDERR_FILE=/nautilus/apps/dagu/logs/myDag/dag-run_20251013_061004Z_0199dc30-b78c-7ce3-9406-c8def329d88e/run_20251013_061004Z_0ef54c/cmd_2.20251013.081004.429.0199dc30.err INVOCATION_ID=a87b8ea529d4478e826aa9c981c1b831 USER=captain SHLVL=0 JOURNAL_STREAM=8:380334880 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin _=/usr/bin/env DAG_DIR_DOCKER=/nautilus/docker_apps

... all the rest of my env variables defined in the .profile (redacted)

Some other checks I ran on my machine directly:

@.***:/nautilus/apps/dagu/dags> id uid=1000(captain) gid=1000(captain) groups=1000(captain)

@.***:/nautilus/apps/dagu/dags> echo $DIR_DOCKER /nautilus/docker_apps

@.***:/nautilus/apps/dagu/dags> echo $DAG_DIR_DOCKER /nautilus/docker_apps

@.***:/nautilus/apps/dagu/dags> ps -elf | grep dagu 4 S captain 3419255 1 0 80 0 - 181113 do_epo Oct10 ? 00:00:00 node /usr/bin/dagu start-all 0 S captain 3419267 3419255 0 80 0 - 316798 futex_ Oct10 ? 00:02:35 /usr/lib/node_modules/dagu/dagu start-all

The definition of dagu systemd service:

[Unit]Description=Dagu Workflow EngineDocumentation=https://docs.dagu.cloud/After=network.targetWants=network-online.target [Service]Type=simpleUser=captainGroup=captainWorkingDirectory=/nautilus/apps/dagu

Main processExecStart=/usr/bin/dagu start-all

Graceful shutdownExecStop=/bin/kill -TERM $MAINPIDTimeoutStopSec=30KillMode=mixedKillSignal=SIGTERM

Restart policyRestart=alwaysRestartSec=10StartLimitInterval=60StartLimitBurst=3

Resource limitsLimitNOFILE=65536LimitNPROC=4096

[Install]WantedBy=multi-user.target

— Reply to this email directly, view it on GitHub https://github.com/dagu-org/dagu/issues/1320#issuecomment-3396049137, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYXFJHRUQXZJXSXLQBBYNL3XNBSBAVCNFSM6AAAAACI7L2PSGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGOJWGA2DSMJTG4 . You are receiving this because you commented.Message ID: @.***>

ghansham avatar Oct 13 '25 07:10 ghansham

I tried to reproduce the issue, but it appears to be working correctly on my end. Am I missing something?

env:
  TEST_DIR: ${TEST_DIR}
steps:
  - name: test
    command: echo "${TEST_DIR}"

~/.zshrc

export TEST_DIR=/tmp

Result (stdout):

/tmp

Ver: 1.22.10 (Started from the Web UI)

Note: sorry for the confusion, but the environment variable filtering feature has not been released yet. It will be in v1.23.0.

yottahmd avatar Oct 13 '25 13:10 yottahmd

@ghansham I did what you asked and this time it worked both with webui and scheduler!

I'm not familiar with systemd services mysteries but it seems that the env is not fully loaded by systemctl here... As a workaround I could launch dagu with nohup, but I could use some help on the systmed service definition for a better solution 🙂

Anyhow thank you guys for your swift responses 🙏

Note: I forgot to say I'm running dagu v1.22.8

thibmart1 avatar Oct 13 '25 18:10 thibmart1

Another point is that if you see the ps -elf | grep dagu output that you attached above, it shows that you are running dagu via node:

node /usr/bin/dagu start-all

I also use system service but I dont depends on system environment variables whose support has been added lately in dagu. I generally set all such variables in base.yaml in DAGU_DIR.

ghansham avatar Oct 13 '25 18:10 ghansham

Is the fact that dagu is running via node a problem of sort? Does the 'direct' run offer more reliability?

thibmart1 avatar Oct 13 '25 20:10 thibmart1

Node is not the source of error.

It is because of systemd service does not load environment variables the way itvis being invoked. Systemd is shell agnostic. We need to use following syntax.

ExecStart=/bin/sh -lc  '/usr/bin/dagu start-all'

or

ExecStart=/bin/bash -lc   '/usr/bin/dagu start-all'

based on the shell you are using.

The -l flag makes the shell invocation a login shell. We need this because only login shells source profile scripts.

@yottahmd may be we can update docs in environment variables section

I found this info from the below link:

https://unix.stackexchange.com/questions/675521/make-systemd-service-inherit-environment-variables-from-etc-profile-d

ghansham avatar Oct 14 '25 01:10 ghansham

Awesome @ghansham, it works like a charm!

I second the update of the doc on this one (and maybe a word or two about the environment variable filtering feature being available in v1.23.0)

thibmart1 avatar Oct 14 '25 06:10 thibmart1

Hi again folks @ghansham @yottahmd

Since version 1.23.0 there is another related issue that arise on my end. Declared env variables in dag definition are well-resolved at runtime (both in CLI and webui) but the ones declared in subdags are not (in the respective subdags commands) (both in CLI and webui).

I reopened the issue but tell me if you prefer a new one.

Edit: I forgot to say I changed the call to subtags from run: to call: as the first was deprecated when upgrading to 1.23 and newer versions, maybe this could help

thibmart1 avatar Nov 05 '25 18:11 thibmart1

Request you to share a sample dag. We will also try one at our end.

ghansham avatar Nov 06 '25 01:11 ghansham

Hi @ghansham

Here is a dag sample:

description: "Various tests on dagu"
tags: [test]

schedule: "0 0 * * *"

env:
  DIR_APP: ${DIR_APP}

steps:
  - name: test app directory
    command: cat ${DIR_APP}/dagu/testfile

handlerOn:
  success:
    run: healthchecks-call
    params:
      - ENDPOINT: 24bd5f99-2976-4c4d-b626-8c943291b835
      - RETURN_CODE: 0
  failure:
    run: healthchecks-call
    params:
      - ENDPOINT: 24bd5f99-2976-4c4d-b626-8c943291b835
      - RETURN_CODE: 1

And the associated subdag:

description: "Send HTTP ping to Healthchecks"
tags: [notif, subtask]

params:
  - ENDPOINT: ""
  - RETURN_CODE: ""

env:
  - DNS_HEALTHCHECKS: ${DNS_HEALTHCHECKS}

steps:
  - command: curl -m 10 --retry 5 https://${DNS_HEALTHCHECKS}/ping/${ENDPOINT}/${RETURN_CODE}

This is the working version in 1.22.10. In 1.23+ I only changed the run command by call and the following behaviour occurs:

  • DIR_APP in the main dag is well-resolved
  • DNS_HEALTHCHECKS in the subdag is not resolved

thibmart1 avatar Nov 07 '25 18:11 thibmart1

Will give a try and dig it out. Thanks

ghansham avatar Nov 08 '25 01:11 ghansham

Hi @ghansham , any news on this issue?

thibmart1 avatar Dec 16 '25 07:12 thibmart1

It should work on the latest release. Can you check please?

ghansham avatar Dec 16 '25 09:12 ghansham

@ghansham I checked on the 1.26.7 version and it still doesn't work for me Added some check commands on the subdag:

  - command: id # to ensure its the same user who calls the dag and the subdag
  - command: echo "DIR_APP=${DIR_APP}" # to check if a env variable from the parent dag is available in the subdag (not sure if it should)
  - command: echo "DNS_HEALTHCHECKS=${DNS_HEALTHCHECKS}" # to check if env variable are resolved within the subdag

Results:

<same user as parent dag>
DIR_APP=
DNS_HEALTHCHECKS=

thibmart1 avatar Dec 22 '25 13:12 thibmart1

Sorry for the delayed in response. Can you mention, where have you set DIR_APP and DNS_HEALTHCHECKS environment variables? bashrc or dagu config files? Is it not working both via cli and UI?

ghansham avatar Dec 22 '25 14:12 ghansham

I set both env variables in a .profile which is loaded with .bashrc It is not working both via cli and UI

thibmart1 avatar Dec 22 '25 14:12 thibmart1

Okay. Let's me replicate and check once

On Mon, Dec 22, 2025, 8:01 PM thibmart1 @.***> wrote:

thibmart1 left a comment (dagu-org/dagu#1320) https://github.com/dagu-org/dagu/issues/1320#issuecomment-3682324484

I set both env variables in a .profile which is loaded with .bashrc It is not working both via cli and UI

— Reply to this email directly, view it on GitHub https://github.com/dagu-org/dagu/issues/1320#issuecomment-3682324484, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYXFJEICRSTP5OQQTHMLGD4C76CPAVCNFSM6AAAAACI7L2PSGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMOBSGMZDINBYGQ . You are receiving this because you were mentioned.Message ID: @.***>

ghansham avatar Dec 22 '25 14:12 ghansham

Please check this:

https://docs.dagu.cloud/writing-workflows/data-variables#system-environment-variable-security

Possible options for you:

  1. The variable names need to be prefixed with LC_
  2. Define them in dotenv and load it in your dag

https://docs.dagu.cloud/writing-workflows/data-variables#dotenv-files

  1. Define them in base.yaml

https://docs.dagu.cloud/writing-workflows/#base-configuration

Please let me know if you are still facing issues.

ghansham avatar Dec 22 '25 14:12 ghansham