Crontab changes

[modem7]

Added ENV vars for crontab, removed old file/way of doing things.

Two new ENV vars have been added:

$CRON
$CRON_COMMAND

If these are unset, they will default to Borg defaults.

Example usage: environment: - CRON=0 5 * * * - CRON_COMMAND=borgmatic --stats -v 0 2>&1

[Skydiver84de]

Cool idea, but what if someone had two lines before in crontab.txt? E.g. one for a daily prune create and one for a monthly check?

[modem7]

I might be able to add something where it can monitor a particular folder for additional files.

I'll investigate further

[grantbevis]

I agree with @Skydiver84de I’d rather allow people to edit their own crontab.txt’s. I implemented something like this at ${work} where in the image I pre set the “top” of the file in /crontab-pre

then I

cat /crontab-pre > crontab-build.txt cat /crontab.txt >> crontab-build.txt crontab crontab-build.txt

Not sure if that would work for us here

[toastie89]

I like the idea. This will be a breaking change and should be part of a major release.

You might go for CRON_10, CRON_22,.. and iterate over those variables.

Is there a special reason why to split Cron and CRON_COMMAND?

[modem7]

I agree with @Skydiver84de I’d rather allow people to edit their own crontab.txt’s.

I've got an idea for this, will post later/tomorrow once I flesh it out!

Is there a special reason why to split Cron and CRON_COMMAND?

Aye, more for basic users vs advanced users.

Ultimately, it makes things simpler and neater for everyone, with less of a chance of messing with default/working settings.

For basic users, it's less likely to mess up a simple cron job they can c+p (from somewhere like crontab) than to mess up the entire line, and it's simpler + neater for those who just want to change the cron time (majority of people).

It also allows for different values available to borgmatic to be changed at will without much hassle for those who wish it.

[modem7]

Would something like this be good? For users advanced enough to figure out further cron jobs, multiline compose files shouldn't be too much of a difficulty.

For single extra requirements:

    environment:
      - CRON=$BORG_CRON
      - CRON_COMMAND=$BORG_CRON_COMMAND
      - EXTRA_CRON=0 5 2 * * mycommand

For multiple (there's lots of ways of doing this, I'm just looking at different ways that look nicer now, but this'll give you the gist):

    environment:
      - CRON=$BORG_CRON
      - CRON_COMMAND=$BORG_CRON_COMMAND
      - |
        EXTRA_CRON=
        0 5 2 * * mycommand1
        0 7 1 * * mycommand2
      - BORG_PASSPHRASE=$BORG_PASSPHRASE

[grantbevis]

I like this

      - |
        EXTRA_CRON=
        0 5 2 * * mycommand1
        0 7 1 * * mycommand2

But I don't like setting the CRON=$BORG_CRON and CRON_COMMAND etc - I'd rather cron actually be a secondary thought in this container and that we write it so people could use it for adhoc backups but we also have the ability to accept a crontab.

Forgive me but I am also not understanding the need to change from a crontab.txt I understand using envars can be neater but it's a big breaking change for lots of users I'd rather not tackle

[modem7]

I mean, it's not difficult to disable cron completely, just don't add and call a file like we're doing by default at the moment, or change the CRON command to different options/change behaviour only if set.

There are plenty of checks that we can do in this case.

I'd rather cron actually be a secondary thought in this container and that we write it so people could use it for adhoc backups but we also have the ability to accept a crontab.

I can easily make a change where if CRON is empty or set to disabled or something, it'd just revert to default behaviour, but going to an ad-hoc model as primary would also be a breaking change.

The way I see it, the reliance on a config file in a docker environment isn't great. It's awful for reproduceability, doesn't scale well, and just generally goes against the ethos of Docker being ephemeral.

It's a necessary evil in many cases of course, but in our case, we can work towards putting as much as we can into an env var, and any further changes users have can be added by some other method, especially since Borgmatic can now parse env vars in configs.

---

So to show how the (current) thinking works:

My current production compose (obviously EXTRA_CRON uncommented for this showcase):

    environment:
      BORG_PASSPHRASE: $BORG_PASSPHRASE
      BORG_SOURCE_1: $BORG_SOURCE_1
      BORG_SOURCE_2: $BORG_SOURCE_2
      BORG_REPO_LOCAL: $BORG_REPO_LOCAL
      BORG_REPO_REMOTE: $BORG_REPO_REMOTE
      BORG_HEALTHCHECK_URL: $BORG_HEALTHCHECK_URL
      CRON: $BORG_CRON
      CRON_COMMAND: $BORG_CRON_COMMAND
      EXTRA_CRON: |-
        0 5 2 * * command1
        0 7 1 * * command2

The output of my latest script is the following:

With $EXTRA_CRON

docker 20.10.16
borgmatic 1.6.6
borg 1.2.1
apprise 1.0.0
Cron job set as: 
0 5 * * * borgmatic --stats -v 1 2>&1
0 5 2 * * command1
0 7 1 * * command2
crond: crond (busybox 1.35.0) started, log level 8

Without $EXTRA_CRON

docker 20.10.16
borgmatic 1.6.6
borg 1.2.1
apprise 1.0.0
Cron job set as: 
0 5 * * * borgmatic --stats -v 1 2>&1
crond: crond (busybox 1.35.0) started, log level 8

Without $CRON_COMMAND

docker 20.10.16
borgmatic 1.6.6
borg 1.2.1
apprise 1.0.0
Cron job set as: 
0 5 * * * borgmatic --stats -v 0 2>&1
crond: crond (busybox 1.35.0) started, log level 8

Without $CRON, $CRON_COMMAND and $EXTRA_CRON

docker 20.10.16
borgmatic 1.6.6
borg 1.2.1
apprise 1.0.0
Cron job set as: 
0 1 * * * borgmatic --stats -v 0 2>&1
crond: crond (busybox 1.35.0) started, log level 8

Cron job set as: is the output of crontab -l

This is the current entry.sh script I'm running in production in my docker-cli build (note the change from /bin/sh to /bin/bash):

#!/bin/bash

# Version variables
dockerver=$(docker --version | grep -Eo '[0-9]+\.[0-9]+\.[0-9]+')
borgver=$(borg --version)
borgmaticver=$(borgmatic --version)
apprisever=$(apprise --version | grep -Eo '[0-9]+\.[0-9]+\.[0-9]+')

# Software versions
echo docker $dockerver
echo borgmatic $borgmaticver
echo $borgver
echo apprise $apprisever

# Generate Cron variables
CRON="${CRON:-"0 1 * * *"}"
CRON_COMMAND="${CRON_COMMAND:-"borgmatic --stats -v 0 2>&1"}"

# Apply cron variables
echo "$CRON $CRON_COMMAND" > /etc/crontabs/root
if [ -v EXTRA_CRON ]
then
   echo "$EXTRA_CRON" >> /etc/crontabs/root
fi

# Current crontab var
crontab=$(crontab -l)

# Output cron settings to console
printf "Cron job set as: \n$crontab\n"

# Start Cron
/usr/sbin/crond -f -L /dev/stdout

---

It should be relatively easy to do the following to make it non-breaking + add new functionality:

• If $CRON is null then < current behaviour >.
• If $CRON is set then < apply $CRON (+ $EXTRA_CRON) values >.

---

The other consideration of the current method (and one of the reasons I decided to implement this route in my environment), is that if you change the crontab.txt file, you still need to restart the container for the changes to be picked up (from what I saw, cron in Docker isn't good enough to pick up changes quickly).

---

Thinking further, there's no reason we can't also add an ad-hoc system, where we can add a value to $CRON which is disabled or false, and it wouldn't apply the cron changes/system at all, allowing people to just run something like: docker exec Borgmatic sh -c "cd && borgmatic --stats -v 1 --files 2>&1" in their terminal to run adhoc backups.

---

Would the following requirements satisfy the above concerns?:

Requirements:

1. If cron is null: Revert to default behaviour (/usr/bin/crontab /etc/borgmatic.d/crontab.txt)
2. If cron is empty or set: Apply cron behaviour
3. If cron is set to disabled, delete cron config (ad-hoc behaviour)

[modem7]

For a TLDR of the above, this script solution should be acceptable to deal with:

1. Adhoc mode as raised by @grantbevis.
2. Custom crontab (both txt and env var) as raised by @Skydiver84de.
3. Env vars as per this PR.
4. Resolving breaking change as raised by @toastie89 without resorting to a major release.

Variables available:

• $CRON
  • Values availables:
    • off
    • disabled
    • false
    • cron times (e.g. 0 1 * * *)
• $CRON_COMMAND
• $EXTRA_CRON

# Disable cron if it's set to disabled.
if [[ "$CRON" =~ ^(false|disabled|off)$ ]]; then
    echo "Disabling cron, removing configuration"
    # crontab -r # quite destructive
    # echo -n > /etc/crontabs/root # Empty config, doesn't look as nice with "crontab -l"
    echo "# Cron disabled" > /etc/crontabs/root
    echo "Cron is now disabled"
# Apply default or custom cron if $CRON is unset or set (not null):
elif [[ -v CRON ]]; then
    CRON="${CRON:-"0 1 * * *"}"
    CRON_COMMAND="${CRON_COMMAND:-"borgmatic --stats -v 0 2>&1"}"
    echo "$CRON $CRON_COMMAND" > /etc/crontabs/root
    echo "Applying custom cron"
# If nothing is set, revert to default behaviour
else
    echo "Applying crontab.txt"
    crontab /etc/borgmatic.d/crontab.txt
fi

# Apply extra cron if it's set
if [ -v EXTRA_CRON ]
then
    echo "$EXTRA_CRON" >> /etc/crontabs/root
fi

# Current crontab var
crontab=$(crontab -l)

# Output cron settings to console
printf "Cron job set as: \n$crontab\n"

The full script would be something similar to:

#!/bin/bash

# Version variables
borgver=$(borg --version)
borgmaticver=$(borgmatic --version)
apprisever=$(apprise --version | grep -Eo '[0-9]+\.[0-9]+\.[0-9]+')

# Software versions
echo borgmatic $borgmaticver
echo $borgver
echo apprise $apprisever

# Disable cron if it's set to disabled.
if [[ "$CRON" =~ ^(false|disabled|off)$ ]]; then
    echo "Disabling cron, removing configuration"
    # crontab -r # quite destructive
    # echo -n > /etc/crontabs/root # Empty config, doesn't look as nice with "crontab -l"
    echo "# Cron disabled" > /etc/crontabs/root
    echo "Cron is now disabled"
# Apply default or custom cron if $CRON is unset or set (not null):
elif [[ -v CRON ]]; then
    CRON="${CRON:-"0 1 * * *"}"
    CRON_COMMAND="${CRON_COMMAND:-"borgmatic --stats -v 0 2>&1"}"
    echo "$CRON $CRON_COMMAND" > /etc/crontabs/root
    echo "Applying custom cron"
# If nothing is set, revert to default behaviour
else
    echo "Applying crontab.txt"
    crontab /etc/borgmatic.d/crontab.txt
fi

# Apply extra cron if it's set
if [ -v EXTRA_CRON ]
then
    echo "$EXTRA_CRON" >> /etc/crontabs/root
fi

# Current crontab var
crontab=$(crontab -l)

# Output cron settings to console
printf "Cron job set as: \n$crontab\n"

# Start Cron
crond -f -L /dev/stdout

Thoughts?

[grantbevis]

This looks good @modem7

[modem7]

This looks good @modem7

@grantbevis sweet, I'll do the necessary changes, and notify when it's pushed.

[modem7]

@grantbevis sorry it took so long - been slammed at work.

This should work as expected with all scenarios in place.

We'll need to think about how to word the update to the readme to include the additional ENV vars at some point:

CRON
CRON_COMMAND
EXTRA_CRON

[modem7]

Looks good @modem7 just to confirm have you tested the full function of this? I don't have the ability to test. But trust your word if you have, also did the docs get updated to follow this change?

Might be worth us having a notice up on the readme

Heya,

Yupper, I can confirm that I've tested as much as I'm able to on this end with standard crontab syntaxes.

The docs have not yet been updated, we probably need to figure out how we're going to be communicating this and what the best way to document it is (format etc).

---

$CRON
Cron timing
Values availables:
off
disabled
false
Example value: 0 1 * * *
Default: already existing crontab.txt or 0 1 * * *

$CRON_COMMAND
Borgmatic command that Cron runs. 
Default: borgmatic --stats -v 0 2>&1
Example value: borgmatic --stats -v 1 2>&1

$EXTRA_CRON
Additional cron lines for advanced usage. 
Default: 
Empty
Example commands:
      EXTRA_CRON: |-
        0 5 2 * * command1
        0 7 1 * * command2

[psi-4ward]

Does docker(-compose) support the array syntax?

  environment:
    - "CRON[]= 0 5 2 * * command1"
    - "CRON[]= 0 5 2 * * command1"

Just an idea

[modem7]

Does docker(-compose) support the array syntax?

  environment:
    - "CRON[]= 0 5 2 * * command1"
    - "CRON[]= 0 5 2 * * command1"

Just an idea

I had a quick look, and on the face of it, it doesn't seem to.

If you're able to get it working, happy to have a look at it!

[psi-4ward]

I had a quick look, and on the face of it, it doesn't seem to.

Looks like this is not supported cause not all shells support array-structs so I would stick to the following syntax

  environment:
    EXTRA_CRON: |-
        0 5 2 * * command1
        0 7 1 * * command2

[modem7]

@grantbevis - Just for an FYI - the merge conflict is caused by updating to latest python 3.11 in this branch.

[grantbevis]

Is this ready to go @modem7 ? Did we need to put a notice on the README?

[modem7]

We need to create some documentation for this but other than that - it's good to go.

Would just a list of variables, defaults + possible options be sufficient? I can whip that up in markdown pretty quick.

[modem7]

Here's a quick set of doco (as markdown support in Github is weird, I've also put a quoted version below in, so it's easier to C+P):

---

Markdown:

Variables

Variable	Description	Default	Example Values
CRON	Cron timing	Existing crontab.txt or 0 1 * * * if no crontab.txt	0 5 * * * off disabled false
CRON_COMMAND	Borgmatic command that Cron runs.	borgmatic --stats -v 0 2>&1	borgmatic --stats -v 2 2>&1
EXTRA_CRON	Additional cron lines for advanced usage.	Empty	See below

Example setup

      CRON: "0 5 * * *"
      CRON_COMMAND: "borgmatic --stats -v 0 2>&1"
      EXTRA_CRON: |-
        0 5 2 * * command1
        0 7 1 * * command2

---

Markdown code block:

# Variables
| Variable | Description | Default | Example Values |
| :----: | --- | --- | --- |
| CRON | Cron timing | Existing crontab.txt or <br> `0 1 * * *` if no crontab.txt | `0 5 * * *` <br> `off` <br> `disabled` <br> `false` |
| CRON_COMMAND | Borgmatic command <br> that Cron runs. | `borgmatic --stats -v 0 2>&1` | `borgmatic --stats -v 2 2>&1` |
| EXTRA_CRON | Additional cron lines <br> for advanced usage. | Empty | ***See below*** |

### Example setup
```shell
      CRON: "0 5 * * *"
      CRON_COMMAND: "borgmatic --stats -v 0 2>&1"
      EXTRA_CRON: |-
        0 5 2 * * command1
        0 7 1 * * command2
```

---