netbird icon indicating copy to clipboard operation
netbird copied to clipboard

NetBird systemd service fails after package update due to missing log directory

Open kimasplund opened this issue 6 months ago • 5 comments

NetBird systemd service fails after package update due to missing log directory

Description

After updating NetBird through the package manager, the systemd service fails to start with exit code 209/STDOUT. The root cause is that the uninstaller removes the /var/log/netbird/ directory during package updates, but the installer/updater does not recreate it.

Environment

  • OS: Ubuntu 25.04 (Linux 6.14.6-061406-generic)
  • NetBird version: 0.44.0
  • Installation method: Package manager (systemd service)

Steps to Reproduce

  1. Have NetBird installed and running
  2. Update NetBird package (e.g., via apt upgrade)
  3. Check service status: sudo systemctl status netbird
  4. Service fails with exit code 209/STDOUT

Expected Behavior

The NetBird service should start successfully after package updates without manual intervention.

Actual Behavior

The service fails to start because the log directory /var/log/netbird/ is missing.

Error Logs

May 23 09:16:50 Kim-UB25-TP systemd[1]: Started netbird.service - A WireGuard-based mesh network that connects your devices into a single private network..
May 23 09:16:50 Kim-UB25-TP (netbird)[28159]: netbird.service: Failed to set up standard output: No such file or directory
May 23 09:16:50 Kim-UB25-TP (netbird)[28159]: netbird.service: Failed at step STDOUT spawning /usr/bin/netbird: No such file or directory
May 23 09:16:50 Kim-UB25-TP systemd[1]: netbird.service: Main process exited, code=exited, status=209/STDOUT
May 23 09:16:50 Kim-UB25-TP systemd[1]: netbird.service: Failed with result 'exit-code'.

Workaround

Manually create the log directory after updates:

sudo mkdir -p /var/log/netbird
sudo chown root:root /var/log/netbird
sudo chmod 755 /var/log/netbird
sudo systemctl restart netbird

Proposed Solution

The package post-install script should ensure the log directory exists:

  1. Check if /var/log/netbird/ exists
  2. If not, create it with appropriate permissions
  3. This should be done in both the installer and updater scripts

Additional Context

The systemd service configuration expects to write logs to /var/log/netbird/client.log as specified in the ExecStart command:

ExecStart=/usr/bin/netbird service run --config /etc/netbird/config.json --log-level info --daemon-addr unix:///var/run/netbird.sock --log-file /var/log/netbird/client.log

This appears to be a packaging issue where the uninstall process (during updates) removes the log directory but the install process doesn't recreate it.

kimasplund avatar May 23 '25 06:05 kimasplund

how is this still a thing guys?! is it just me his thing affects? i need to recreate the directory on every update...

kimasplund avatar Jun 10 '25 13:06 kimasplund

@kimasplund, it is not a common issue. There might be something specific about how the system is set.

Can you share more details on your OS? Is there a mounted log directory? Is app Armor or SELinux enabled? any custom systemd setup?

mlsmaycon avatar Jun 10 '25 13:06 mlsmaycon

nope. nothing special. just default Ubuntu 25.04 on my laptop.

System Details Report


Report details

  • Date generated: 2025-06-10 16:35:08

Hardware Information:

  • Hardware Model: Lenovo ThinkPad P16 Gen 2
  • Memory: 64.0 GiB
  • Processor: 13th Gen Intel® Core™ i9-13980HX × 32
  • Graphics: Intel® Graphics (RPL-S)
  • Graphics 1: NVIDIA RTX 3500 Ada Generation Laptop GPU
  • Disk Capacity: 4.0 TB

Software Information:

  • Firmware Version: N3TET57W (1.57 )
  • OS Name: Ubuntu 25.04
  • OS Build: (null)
  • OS Type: 64-bit
  • GNOME Version: 48
  • Windowing System: Wayland
  • Kernel Version: Linux 6.14.8-061408-generic

tmpfs on /var/log type tmpfs (rw,noatime,size=65536k,mode=755,inode64)

kimasplund avatar Jun 10 '25 13:06 kimasplund

but sure is annoying as... to need to constantly recreate the log folder. come on guys. just add the function to check it on upgrade. even provided you with the patch for it ;D

kimasplund avatar Jun 11 '25 15:06 kimasplund

we are running a few tests on your PR and will approve it when they are done

mlsmaycon avatar Jun 11 '25 15:06 mlsmaycon

I've also just experienced this. I think the difference on my system is that I'm running on OpenMediaVault on a Raspberry Pi that has a SD card as its OS storage, so OMV by default uses a flashmemory-aware plugin that keeps noisy writes from going straight to the SD card in a memory FS temporary, flushing it on reboot. I wonder if that's part of the problem in this particular case?

webash avatar Oct 26 '25 10:10 webash

Just wanted to add my experience to this issue. I'm trying to add Netbird to my custom atomic image based on Bazzite. The issue with installing it on an image-based OS is that during install/build time, the eventual /var of the future running system doesn't exist. Only attempting to create the log folder during install is causing the problems in this case.

The main issue boils down to the use of StandardOutput and StandardError in the systemd service file. Those lines require the target folder to exist, and the whole service dies if it doesn't. If those are taken out, the log folder gets created correctly when the ExecStart command runs.

To be honest, I'm not totally sure the reasoning behind using those directives at all. Systemd/Journald handle stdout/stderr quite nicely already, so it seems odd to me to force the outputs to those log files unless absolutely necessary.

reyemxela avatar Nov 07 '25 21:11 reyemxela