random_sleep is ignored on machines using systemd

[baldurmen]

Hi!

It seems that on Debian machines running systemd (at least on Debian Buster), the random_sleep parameter is ignored.

Indeed, apt transitioned from a cronjob to a systemd timer. /etc/cron.daily/apt-compat now starts with:

# Systemd systems use a systemd timer unit which is preferable to
# run. We want to randomize the apt update and unattended-upgrade
# runs as much as possible to avoid hitting the mirrors all at the
# same time. The systemd time is better at this than the fixed
# cron.daily time
if [ -d /run/systemd/system ]; then
    exit 0
fi

This means the cronjob exits if systemd is running. If that's the case, the machine uses /lib/systemd/system/apt-daily.timer instead to manage automated updates. On Buster, this file looks like:

[Unit]
Description=Daily apt download activities

[Timer]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true

[Install]
WantedBy=timers.target

It also seems RandomizedDelaySec is hard coded. I don't see a bug on apt for this issue, but I'll open one shortly.

To fix this issue, this module would need to modify the timer using the value passed to random_sleep and reload systemd daemons.

Cheers,

[igalic]

can someone confirm if this is the same for Ubuntu?

[baldurmen]

The upstream bug I opened.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965997

I'm 90% sure this is also the case on Ubuntu, as the main apt dev also does a lot of Ubuntu work :)

[ekohl]

The benefit of letting systemd do this is that you don't have a process just sleeping for some time. Instead, systemd will spawn the process at some time. To modify this, you would indeed need a systemd drop in.

[igalic]

mumbles something about augeas

[ekohl]

That's IMHO a bad idea. You don't want to modify /lib/systemd/system/apt-daily.timer since that's managed by the OS. Augeas can probably manage a drop in, but you still want that to live in /etc/systemd.

[baldurmen]

All in all, this should really be fixed in apt, everything else is a hot glue patch.

That doesn't mean people using this module shouldn't be aware of the issue though :)

Temporarily deprecating random_sleep until the upstream bug is fixed is also a valid option IMO.

[ekohl]

I think short term it's sufficient to change the parameter documentation to mention it's only effective on non-systemd systems, linking to the upstream bug.

[baldurmen]

Well, the apt upstream closed the bug report with a wontfix, so if we want this to work, we'll have to make it work ourselves.

Upstream says:

No, please configure the timer with a drop-in.

From that I understand we should be replacing /lib/systemd/system/apt-daily.timer with a templated version or change the value with something like file_line.

[ekohl]

From that I understand we should be replacing /lib/systemd/system/apt-daily.timer with a templated version or change the value with something like file_line.

No, this is not what a systemd drop-in is. You can easily see how it works by running systemctl edit apt-daily.timer and pasting the following:

[Timer]
RandomizedDelaySec=8h

Then you can run systemctl cat apt-daily.timer to see the result.

See man systemd.unit

https://github.com/camptocamp/puppet-systemd/blob/master/manifests/dropin_file.pp would be a type that easily allows you to create one.

[ekohl]

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965997 was closed and also says a systemd drop in is the right way to go.

[anarcat]

I'm not sure I understand the issue here: is it that:

1. you want to be able to tweak that parameter under systemd and you can't? or that:
2. you worry that we have a deprecated parameter around and we should remove it or warn about it?

I am not sure why you'd need to change that parameter in the first place, in any case, so I'm genuinely curious about the use case here.

[baldurmen]

@anarcat This is a pretty old issue, but the use case is not to have systems update at a random time and instead run u-u at the same time. This simplifies planned reboots for me, as I can be sure all the systems I manage have been upgraded when time comes to reboot them (instead of having them upgrade at different times).

As per the issue:

1. It would be nice to be able to do that yes. That would require systemd drop-ins.
2. This parameter is currently deprecated (at least, for all systems using systemd...) but the documentation doesn't reflect this.

[anarcat]

On 2023-12-15 10:57:17, Louis-Philippe Véronneau wrote:

the use case is not to have systems update at a random time and instead run u-u at the same time

[...]

See, I'm not sure that's a good idea in the first place. We don't want to have fleets all checking in at the same time, that's a bad idea for numerous reasons, the first of which is overload on the mirror network, but it also makes your systems possibly more brittle, as a botched upgrade will scrap the entire fleet...

As per the issue:

1. It would be nice to be able to do that yes. That would require systemd drop-ins.

But does that need to be in this module? You could deploy that drop-in yourself...

2. This parameter is currently deprecated (at least, for all systems using systemd...) but the documentation doesn't reflect this.

That, on the other hand, seem like something that should be fixed. :)