Systemd service - fail on boot - have to add dependencies

[FGIKCM]

Hello,

With your minimal example of Systemd service (https://github.com/nextcloud/notify_push), it failed to start at boot on my host, as needed dependencies where not started.

Example:

Aug 20 10:07:55 myserver systemd[1]: Started The PHP 7.4 FastCGI Process Manager.
Aug 20 10:07:55 myserver systemd[1]: Started Nextcloud - Push daemon for Nextcloud clients.
Aug 20 10:07:55 myserver notify_push[890]: Error:
Aug 20 10:07:55 myserver notify_push[890]:    0: #033[91mFailed to connect to Nextcloud database#033[0m
Aug 20 10:07:55 myserver notify_push[890]:    1: #033[91merror communicating with the server: No such file or directory (os error 2)#033[0m
Aug 20 10:07:55 myserver notify_push[890]:    2: #033[91mNo such file or directory (os error 2)#033[0m
Aug 20 10:07:55 myserver notify_push[890]: Location:
Aug 20 10:07:55 myserver notify_push[890]:    #033[35m/tmp/krankerl.AjpQJOOqCJm6/notify_push/src/storage_mapping.rs#033[0m:#033[35m57#033[0m
Aug 20 10:07:55 myserver notify_push[890]: Backtrace omitted.
Aug 20 10:07:55 myserver notify_push[890]: Run with RUST_BACKTRACE=1 environment variable to display it.
Aug 20 10:07:55 myserver notify_push[890]: Run with RUST_BACKTRACE=full to include source snippets.
Aug 20 10:07:55 myserver systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Aug 20 10:07:55 myserver systemd[1]: notify_push.service: Failed with result 'exit-code'.

To fix it, I added After=, to be sure everything else is started before (Apache, PHP-FPM, MySQL, Redis...)

[Unit]
Description = Push daemon for Nextcloud clients
Documentation=https://github.com/nextcloud/notify_push
After=apache2.service php7.4-fpm.service mysql.service redis.service

You should perhaps add some clues on this on your wiki page.

Best regards

[solracsf]

I'm having the same error, but adding After didn't help.

Error:
   0: #033[91mFailed to connect to Nextcloud database#033[0m
   1: #033[91merror communicating with the server: No such file or directory (os error 2)#033[0m
   2: #033[91mNo such file or directory (os error 2)#033[0m
Location:
   #033[35m/tmp/krankerl.AjpQJOOqCJm6/notify_push/src/storage_mapping.rs#033[0m:#033[35m57#033[0m
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  #033[96m                               ⋮ 1 frame hidden ⋮                               #033[0m
   2: #033[91m<E as eyre::context::ext::StdError>::ext_report#033[0m#033[90m::hcacf9c76931b0615#033[0m
      at #033[35m<unknown source file>#033[0m:#033[35m<unknown line>#033[0m
   3: #033[91mnotify_push::main::{{closure}}#033[0m#033[90m::h7dd60ba34edc8db9#033[0m
      at #033[35m<unknown source file>#033[0m:#033[35m<unknown line>#033[0m
   4: #033[91mnotify_push::main#033[0m#033[90m::ha347dbf59e83ceb0#033[0m
      at #033[35m<unknown source file>#033[0m:#033[35m<unknown line>#033[0m
   5: #033[32mstd::sys_common::backtrace::__rust_begin_short_backtrace#033[0m#033[90m::hf0388764817a5d49#033[0m
      at #033[35m<unknown source file>#033[0m:#033[35m<unknown line>#033[0m
   6: #033[32mmain#033[0m#033[90m<unknown>#033[0m
      at #033[35m<unknown source file>#033[0m:#033[35m<unknown line>#033[0m
Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Run with RUST_BACKTRACE=full to include source snippets.

Tests are OK:

php occ notify_push:self-test
✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
✓ push server can connect to the Nextcloud server
✓ push server is a trusted proxy
✓ push server is running the same version as the app

[FGIKCM]

I'm having the same error, but adding After didn't help.

Did you do following command to take changes in acocunt?

systemctl daemon-reload

(If not, that's still the old config that is active)

[solracsf]

Yes I did, as always ;)

[FGIKCM]

Ah... I have no more ideas... perhaps some other dependencies, like local fs (local-fs.target) or networking (network.target)? But as those targets should be started by Apache and so, I don't think it's the answer. Or on your system, a service name is not exactly the same, and for example you do not have php7.4-fpm, but php7.3-fpm, something like that? (but I guess you already checked too)

To go further, I also tried to declare env variables COLORBT_SHOW_HIDDEN and RUST_BACKTRACE, to see if we have more outputs in the syslog, but with no luck...

I added (with no effect):

[Service]
Environment = COLORBT_SHOW_HIDDEN=1
Environment = RUST_BACKTRACE=full

So, I'm afraid I have no other ideas :-/

[DanScharon]

After=network.target is needed as an absolute minimum, otherwise startup on boot fails, even on hosts where only notify_push is installed (and no other dependent elements).

[solracsf]

I have After=network-online.target

[jkhsjdhjs]

Adding After=postgresql.service fixed this for me. I added Requires=postgresql.service as well, because notify_push has a hard dependency on the database and should fail if the database fails, so it reflects the dependency better.

The postgresql.service file for my distro uses Type=notify, so the unit is only considered running when the postgresql server tells systemd so. This is important for the After dependency, because the UNIX socket or TCP port need to be created/open for notify_push to work. You can check if that's the case for the service file of the database server on your distro as well.

Other than that, if you're using a TCP port to connect to your database server, you can try switching to the UNIX socket instead, since using the TCP port may require network.target or network-online.target. However, this is just a guess, I'm not sure whether this is really the case.

[reos-rcrozier]

I had the same problem, at least, starting was failing and I see the following:

$ systemctl status notify_push.service 
● notify_push.service - Push daemon for Nextcloud clients
     Loaded: loaded (/etc/systemd/system/notify_push.service; disabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/notify_push.service.d
             └─override.conf
     Active: inactive (dead)
       Docs: https://github.com/nextcloud/notify_push

Jan 16 00:54:55 ed-mh-x86001 systemd[1]: /etc/systemd/system/notify_push.service:7: Invalid environment assignment, ignoring: Change
Jan 16 00:54:55 ed-mh-x86001 systemd[1]: /etc/systemd/system/notify_push.service:7: Invalid environment assignment, ignoring: if
Jan 16 00:54:55 ed-mh-x86001 systemd[1]: /etc/systemd/system/notify_push.service:7: Invalid environment assignment, ignoring: you
Jan 16 00:54:55 ed-mh-x86001 systemd[1]: /etc/systemd/system/notify_push.service:7: Invalid environment assignment, ignoring: already
Jan 16 00:54:55 ed-mh-x86001 systemd[1]: /etc/systemd/system/notify_push.service:7: Invalid environment assignment, ignoring: have
Jan 16 00:54:55 ed-mh-x86001 systemd[1]: /etc/systemd/system/notify_push.service:7: Invalid environment assignment, ignoring: something
Jan 16 00:54:55 ed-mh-x86001 systemd[1]: /etc/systemd/system/notify_push.service:7: Invalid environment assignment, ignoring: running
Jan 16 00:54:55 ed-mh-x86001 systemd[1]: /etc/systemd/system/notify_push.service:7: Invalid environment assignment, ignoring: on
Jan 16 00:54:55 ed-mh-x86001 systemd[1]: /etc/systemd/system/notify_push.service:7: Invalid environment assignment, ignoring: this
Jan 16 00:54:55 ed-mh-x86001 systemd[1]: /etc/systemd/system/notify_push.service:7: Invalid environment assignment, ignoring: port

adding

After=apache2.service php7.4-fpm.service mysql.service redis.service

did not solve it for me

I tried the bazooka approach with

[Unit]
After=network-online.target apache2.service mariadb.service redis.service
Requires=apache2.service mariadb.service redis.service

[Service]
ExecStartPre=/usr/bin/sleep 30

This didn't work either.

Starting manually with systemctl start notify_push.service works fine

[Sir-Photch]

After=apache2.service php7.4-fpm.service mysql.service redis.service

Thanks for investigating this. I can confirm that this solves the issue of notify_push failing after reboot of the system.

[joshtrichards]

Also see #194 & #175 for some ideas for addressing.