StartUnixWorker(): watch forked child via waitpid(), not SIGCHLD handler

[Al2Klimov]

Before

On SIGCHLD from the forked worker the umbrella process sets a failure flag. StartUnixWorker() recognises that and does waitpid(), failure message, etc.. On OpenBSD we can't tell the signal source, so we always set the failure flag. That's not how our IPC shall work, that breaks the IPC sooner or later.

After

No SIGCHLD handling and no failure flag setting. Instead StartUnixWorker()'s wait loop uses waitpid(x,y,WNOHANG) to avoid false positives while watching the forked worker.

fixes #9481

[Al2Klimov]

Tests

Initial config is invalid

➜  icinga2 git:(9481) prefix/sbin/icinga2 daemon -c my.conf
[2022-09-07 12:23:36 +0200] information/cli: Icinga application loader (version: v2.13.0-427-g22bfcf9ac; debug)
[2022-09-07 12:23:36 +0200] information/cli: Loading configuration file(s).
[2022-09-07 12:23:36 +0200] critical/config: Error: Unterminated string literal
Location: in my.conf: 1:0-2:0
my.conf(1): "LOLCAT
            ^^^^^^^^
my.conf(2):

[2022-09-07 12:23:36 +0200] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.
➜  icinga2 git:(9481) echo $?
1
➜  icinga2 git:(9481)

Fails as expected.

Config is valid/invalid/valid/invalid/valid/valid/...

➜  icinga2 git:(9481) prefix/sbin/icinga2 daemon -c my.conf
[2022-09-07 12:26:23 +0200] information/cli: Icinga application loader (version: v2.13.0-427-g22bfcf9ac; debug)
[2022-09-07 12:26:23 +0200] information/cli: Loading configuration file(s).
[2022-09-07 12:26:23 +0200] information/ConfigItem: Committing config item(s).
[2022-09-07 12:26:23 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-09-07 12:26:23 +0200] information/ScriptGlobal: Dumping variables to file '/Users/aklimov/NET/WS/icinga2/prefix/var/cache/icinga2/icinga2.vars'
[2022-09-07 12:26:23 +0200] information/ConfigItem: Triggering Start signal for config items
[2022-09-07 12:26:23 +0200] information/ConfigItem: Activated all objects.
[2022-09-07 12:27:06 +0200] information/Application: Got reload command: Starting new instance.
[2022-09-07 12:27:06 +0200] information/cli: Loading configuration file(s).
[2022-09-07 12:27:06 +0200] critical/config: Error: Unterminated string literal
Location: in my.conf: 1:0-2:0
my.conf(1): "LOLCAT
            ^^^^^^^^
my.conf(2):

[2022-09-07 12:27:06 +0200] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.
[2022-09-07 12:27:06 +0200] critical/Application: Found error in config: reloading aborted
[2022-09-07 12:30:03 +0200] information/Application: Got reload command: Starting new instance.
[2022-09-07 12:30:03 +0200] information/cli: Loading configuration file(s).
[2022-09-07 12:30:03 +0200] information/ConfigItem: Committing config item(s).
[2022-09-07 12:30:03 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-09-07 12:30:03 +0200] information/ScriptGlobal: Dumping variables to file '/Users/aklimov/NET/WS/icinga2/prefix/var/cache/icinga2/icinga2.vars'
[2022-09-07 12:30:03 +0200] information/Application: Reload done, old process shutting down. Child process with PID '65746' is taking over.
[2022-09-07 12:30:03 +0200] information/Application: Received request to shut down.
[2022-09-07 12:30:04 +0200] information/Application: Shutting down...
[2022-09-07 12:30:04 +0200] information/ConfigObject: Dumping program state to file '/Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/icinga2.state'
[2022-09-07 12:30:04 +0200] information/IcingaApplication: Icinga has shut down.
[2022-09-07 12:30:04 +0200] information/ConfigObject: Restoring program state from file '/Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/icinga2.state'
[2022-09-07 12:30:04 +0200] information/ConfigObject: Restored 3 objects. Loaded 0 new objects without state.
[2022-09-07 12:30:04 +0200] information/ConfigItem: Triggering Start signal for config items
[2022-09-07 12:30:04 +0200] information/ConfigItem: Activated all objects.
[2022-09-07 12:30:29 +0200] information/Application: Got reload command: Starting new instance.
[2022-09-07 12:30:29 +0200] information/cli: Loading configuration file(s).
[2022-09-07 12:30:29 +0200] critical/config: Error: Unterminated string literal
Location: in my.conf: 1:0-2:0
my.conf(1): "LOLCAT
            ^^^^^^^^
my.conf(2):

[2022-09-07 12:30:29 +0200] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.
[2022-09-07 12:30:29 +0200] critical/Application: Found error in config: reloading aborted
[2022-09-07 12:30:43 +0200] information/Application: Got reload command: Starting new instance.
[2022-09-07 12:30:43 +0200] information/cli: Loading configuration file(s).
[2022-09-07 12:30:43 +0200] information/ConfigItem: Committing config item(s).
[2022-09-07 12:30:43 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-09-07 12:30:43 +0200] information/ScriptGlobal: Dumping variables to file '/Users/aklimov/NET/WS/icinga2/prefix/var/cache/icinga2/icinga2.vars'
[2022-09-07 12:30:43 +0200] information/Application: Reload done, old process shutting down. Child process with PID '65840' is taking over.
[2022-09-07 12:30:43 +0200] information/Application: Received request to shut down.
[2022-09-07 12:30:43 +0200] information/Application: Shutting down...
[2022-09-07 12:30:43 +0200] information/ConfigObject: Dumping program state to file '/Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/icinga2.state'
[2022-09-07 12:30:43 +0200] information/IcingaApplication: Icinga has shut down.
[2022-09-07 12:30:43 +0200] information/ConfigObject: Restoring program state from file '/Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/icinga2.state'
[2022-09-07 12:30:43 +0200] information/ConfigObject: Restored 3 objects. Loaded 0 new objects without state.
[2022-09-07 12:30:43 +0200] information/ConfigItem: Triggering Start signal for config items
[2022-09-07 12:30:43 +0200] information/ConfigItem: Activated all objects.
[2022-09-07 12:30:47 +0200] information/Application: Got reload command: Starting new instance.
[2022-09-07 12:30:47 +0200] information/cli: Loading configuration file(s).
[2022-09-07 12:30:47 +0200] information/ConfigItem: Committing config item(s).
[2022-09-07 12:30:47 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-09-07 12:30:47 +0200] information/ScriptGlobal: Dumping variables to file '/Users/aklimov/NET/WS/icinga2/prefix/var/cache/icinga2/icinga2.vars'
[2022-09-07 12:30:47 +0200] information/Application: Reload done, old process shutting down. Child process with PID '65860' is taking over.
[2022-09-07 12:30:47 +0200] information/Application: Received request to shut down.
[2022-09-07 12:30:48 +0200] information/Application: Shutting down...
[2022-09-07 12:30:48 +0200] information/ConfigObject: Dumping program state to file '/Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/icinga2.state'
[2022-09-07 12:30:48 +0200] information/IcingaApplication: Icinga has shut down.
[2022-09-07 12:30:48 +0200] information/ConfigObject: Restoring program state from file '/Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/icinga2.state'
[2022-09-07 12:30:48 +0200] information/ConfigObject: Restored 3 objects. Loaded 0 new objects without state.
[2022-09-07 12:30:48 +0200] information/ConfigItem: Triggering Start signal for config items
[2022-09-07 12:30:48 +0200] information/ConfigItem: Activated all objects.
[2022-09-07 12:30:52 +0200] information/Application: Got reload command: Starting new instance.
[2022-09-07 12:30:52 +0200] information/cli: Loading configuration file(s).
[2022-09-07 12:30:52 +0200] information/ConfigItem: Committing config item(s).
[2022-09-07 12:30:52 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-09-07 12:30:52 +0200] information/ScriptGlobal: Dumping variables to file '/Users/aklimov/NET/WS/icinga2/prefix/var/cache/icinga2/icinga2.vars'
[2022-09-07 12:30:52 +0200] information/Application: Reload done, old process shutting down. Child process with PID '65874' is taking over.
[2022-09-07 12:30:52 +0200] information/Application: Received request to shut down.
[2022-09-07 12:30:52 +0200] information/Application: Shutting down...
[2022-09-07 12:30:52 +0200] information/ConfigObject: Dumping program state to file '/Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/icinga2.state'
[2022-09-07 12:30:52 +0200] information/IcingaApplication: Icinga has shut down.
[2022-09-07 12:30:52 +0200] information/ConfigObject: Restoring program state from file '/Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/icinga2.state'
[2022-09-07 12:30:52 +0200] information/ConfigObject: Restored 3 objects. Loaded 0 new objects without state.
[2022-09-07 12:30:52 +0200] information/ConfigItem: Triggering Start signal for config items
[2022-09-07 12:30:52 +0200] information/ConfigItem: Activated all objects.

Everything still works.

➜  icinga2 git:(9481) pstree |grep icinga2
 |   |   \-+= 65440 aklimov /Users/aklimov/NET/WS/icinga2/prefix/lib/icinga2/sbin/icinga2 daemon -c my.conf
 |   |     \-+- 65874 aklimov /Users/aklimov/NET/WS/icinga2/prefix/lib/icinga2/sbin/icinga2 daemon -c my.conf
 |   |       \--- 65876 aklimov /Users/aklimov/NET/WS/icinga2/prefix/lib/icinga2/sbin/icinga2 daemon -c my.conf
 |   |   \--- 65953 aklimov grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn icinga2
➜  icinga2 git:(9481)

No zombies.

OpenBSD before/after

(/usr/ports/net/icinga/core2 is still building...)

[Al2Klimov]

The first of the three commits is enough to fix the problem. I wanted to test it on OpenBSD myself, but make seems not to take my port patch into account. :(

[Al2Klimov]

OpenBSD before

aklimov-openbsd# icinga2 daemon
[2022-09-08 12:44:16 +0200] information/cli: Icinga application loader (version: r2.13.2-1)
[2022-09-08 12:44:16 +0200] information/cli: Loading configuration file(s).
[2022-09-08 12:44:16 +0200] information/ConfigItem: Committing config item(s).
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 1 Host.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 244 CheckCommands.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 3 ServiceGroups.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 3 HostGroups.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 12 Notifications.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 1 User.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 11 Services.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 3 Zones.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-09-08 12:44:17 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-09-08 12:44:17 +0200] information/ConfigObject: Restoring program state from file '/var/lib/icinga2/icinga2.state'
[2022-09-08 12:44:17 +0200] information/ConfigObject: Restored 292 objects. Loaded 0 new objects without state.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Triggering Start signal for config items
[2022-09-08 12:44:17 +0200] information/FileLogger: 'debug-file' started.
[2022-09-08 12:44:17 +0200] information/FileLogger: 'main-log' started.
[2022-09-08 12:44:17 +0200] information/NotificationComponent: 'notification' started.
[2022-09-08 12:44:17 +0200] information/CheckerComponent: 'checker' started.
[2022-09-08 12:44:17 +0200] information/ConfigItem: Activated all objects.
[2022-09-08 12:44:20 +0200] information/Application: Got reload command: Starting new instance.
[2022-09-08 12:44:20 +0200] information/cli: Loading configuration file(s).
[2022-09-08 12:44:20 +0200] information/ConfigItem: Committing config item(s).
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 1 Host.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 244 CheckCommands.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 3 ServiceGroups.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 3 HostGroups.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 12 Notifications.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 1 User.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 11 Services.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 3 Zones.
[2022-09-08 12:44:21 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-09-08 12:44:21 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-09-08 12:44:21 +0200] information/Application: Reload done, old process shutting down. Child process with PID '7687' is taking over.
[2022-09-08 12:44:21 +0200] information/Application: Received request to shut down.
[2022-09-08 12:44:22 +0200] information/Application: Shutting down...
[2022-09-08 12:44:22 +0200] information/CheckerComponent: 'checker' stopped.
[2022-09-08 12:44:22 +0200] information/NotificationComponent: 'notification' stopped.
[2022-09-08 12:44:22 +0200] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2022-09-08 12:44:22 +0200] information/IcingaApplication: Icinga has shut down.
[2022-09-08 12:44:22 +0200] information/ConfigObject: Restoring program state from file '/var/lib/icinga2/icinga2.state'
[2022-09-08 12:44:22 +0200] information/ConfigObject: Restored 292 objects. Loaded 0 new objects without state.
[2022-09-08 12:44:22 +0200] information/ConfigItem: Triggering Start signal for config items
[2022-09-08 12:44:22 +0200] information/FileLogger: 'debug-file' started.
[2022-09-08 12:44:22 +0200] information/FileLogger: 'main-log' started.
[2022-09-08 12:44:22 +0200] information/NotificationComponent: 'notification' started.
[2022-09-08 12:44:22 +0200] information/CheckerComponent: 'checker' started.
[2022-09-08 12:44:22 +0200] information/ConfigItem: Activated all objects.
[2022-09-08 12:44:27 +0200] information/Application: Got reload command: Starting new instance.
[2022-09-08 12:44:27 +0200] information/cli: Loading configuration file(s).
[2022-09-08 12:44:27 +0200] information/ConfigItem: Committing config item(s).
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 1 Host.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 244 CheckCommands.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 3 ServiceGroups.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 3 HostGroups.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 12 Notifications.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 1 User.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 11 Services.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 3 Zones.
[2022-09-08 12:44:28 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-09-08 12:44:28 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-09-08 12:45:22 +0200] information/ConfigObjectUtility: Created and activated object 'aklimov-openbsd.my.domain!load!2334820e-0552-4965-953c-78772614e7db' of type 'Downtime'.
[2022-09-08 12:45:22 +0200] information/Downtime: Added downtime 'aklimov-openbsd.my.domain!load!2334820e-0552-4965-953c-78772614e7db' between '2022-09-09 02:00:00' and '2022-09-09 03:00:00', author: 'icingaadmin', fixed
[2022-09-08 12:45:49 +0200] information/Checkable: Checkable 'aklimov-openbsd.my.domain!http' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2022-09-08 12:45:49 +0200] information/Notification: Sending 'Problem' notification 'aklimov-openbsd.my.domain!http!mail-icingaadmin' for user 'icingaadmin'
[2022-09-08 12:45:49 +0200] information/Notification: Completed sending 'Problem' notification 'aklimov-openbsd.my.domain!http!mail-icingaadmin' for checkable 'aklimov-openbsd.my.domain!http' and user 'icingaadmin' using command 'mail-service-notification'.
[2022-09-08 12:45:58 +0200] information/Checkable: Checkable 'aklimov-openbsd.my.domain!swap' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2022-09-08 12:45:58 +0200] information/Notification: Sending 'Problem' notification 'aklimov-openbsd.my.domain!swap!mail-icingaadmin' for user 'icingaadmin'
[2022-09-08 12:45:58 +0200] information/Notification: Completed sending 'Problem' notification 'aklimov-openbsd.my.domain!swap!mail-icingaadmin' for checkable 'aklimov-openbsd.my.domain!swap' and user 'icingaadmin' using command 'mail-service-notification'.

No reaction to kill -HUP $UMBRELLA_PID anymore.

OpenBSD after

Actually the first commit is not enough. Don’t ask me how I got all three (monkey-)patched into my ports.

aklimov-openbsd# icinga2 daemon
[2022-09-08 17:16:11 +0200] information/cli: Icinga application loader (version: r2.13.2-1)
[2022-09-08 17:16:11 +0200] information/cli: Loading configuration file(s).
[2022-09-08 17:16:11 +0200] information/ConfigItem: Committing config item(s).
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 12 Notifications.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 3 ServiceGroups.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 3 Zones.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 11 Services.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 1 User.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 1 Host.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 1 Downtime.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 3 HostGroups.
[2022-09-08 17:16:11 +0200] information/ConfigItem: Instantiated 244 CheckCommands.
[2022-09-08 17:16:11 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-09-08 17:16:12 +0200] information/ConfigObject: Restoring program state from file '/var/lib/icinga2/icinga2.state'
[2022-09-08 17:16:12 +0200] information/ConfigObject: Restored 293 objects. Loaded 0 new objects without state.
[2022-09-08 17:16:12 +0200] information/ConfigItem: Triggering Start signal for config items
[2022-09-08 17:16:12 +0200] information/FileLogger: 'debug-file' started.
[2022-09-08 17:16:12 +0200] information/FileLogger: 'main-log' started.
[2022-09-08 17:16:12 +0200] information/NotificationComponent: 'notification' started.
[2022-09-08 17:16:12 +0200] information/CheckerComponent: 'checker' started.
[2022-09-08 17:16:12 +0200] information/ConfigItem: Activated all objects.
[2022-09-08 17:16:17 +0200] information/Notification: Sending reminder 'Problem' notification 'aklimov-openbsd.my.domain!http!mail-icingaadmin' for user 'icingaadmin'
[2022-09-08 17:16:17 +0200] information/Notification: Sending reminder 'Problem' notification 'aklimov-openbsd.my.domain!swap!mail-icingaadmin' for user 'icingaadmin'
[2022-09-08 17:16:17 +0200] information/Notification: Completed sending 'Problem' notification 'aklimov-openbsd.my.domain!http!mail-icingaadmin' for checkable 'aklimov-openbsd.my.domain!http' and user 'icingaadmin' using command 'mail-service-notification'.
[2022-09-08 17:16:17 +0200] information/Notification: Completed sending 'Problem' notification 'aklimov-openbsd.my.domain!swap!mail-icingaadmin' for checkable 'aklimov-openbsd.my.domain!swap' and user 'icingaadmin' using command 'mail-service-notification'.
[2022-09-08 17:16:17 +0200] information/Application: Got reload command: Starting new instance.
[2022-09-08 17:16:17 +0200] information/cli: Loading configuration file(s).
[2022-09-08 17:16:17 +0200] information/ConfigItem: Committing config item(s).
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 12 Notifications.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 3 ServiceGroups.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 3 Zones.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 11 Services.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 1 User.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 1 Host.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 1 Downtime.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 3 HostGroups.
[2022-09-08 17:16:18 +0200] information/ConfigItem: Instantiated 244 CheckCommands.
[2022-09-08 17:16:18 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-09-08 17:16:18 +0200] information/Application: Reload done, old process shutting down. Child process with PID '56767' is taking over.
[2022-09-08 17:16:18 +0200] information/Application: Received request to shut down.
[2022-09-08 17:16:19 +0200] information/Application: Shutting down...
[2022-09-08 17:16:19 +0200] information/CheckerComponent: 'checker' stopped.
[2022-09-08 17:16:19 +0200] information/NotificationComponent: 'notification' stopped.
[2022-09-08 17:16:19 +0200] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2022-09-08 17:16:19 +0200] information/IcingaApplication: Icinga has shut down.
[2022-09-08 17:16:19 +0200] information/ConfigObject: Restoring program state from file '/var/lib/icinga2/icinga2.state'
[2022-09-08 17:16:19 +0200] information/ConfigObject: Restored 293 objects. Loaded 0 new objects without state.
[2022-09-08 17:16:19 +0200] information/ConfigItem: Triggering Start signal for config items
[2022-09-08 17:16:19 +0200] information/FileLogger: 'debug-file' started.
[2022-09-08 17:16:19 +0200] information/FileLogger: 'main-log' started.
[2022-09-08 17:16:19 +0200] information/NotificationComponent: 'notification' started.
[2022-09-08 17:16:19 +0200] information/CheckerComponent: 'checker' started.
[2022-09-08 17:16:19 +0200] information/ConfigItem: Activated all objects.
[2022-09-08 17:16:26 +0200] information/Application: Got reload command: Starting new instance.
[2022-09-08 17:16:26 +0200] information/cli: Loading configuration file(s).
[2022-09-08 17:16:26 +0200] information/ConfigItem: Committing config item(s).
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 12 Notifications.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 3 ServiceGroups.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 3 Zones.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 11 Services.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 1 User.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 1 Host.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 1 Downtime.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 3 HostGroups.
[2022-09-08 17:16:26 +0200] information/ConfigItem: Instantiated 244 CheckCommands.
[2022-09-08 17:16:26 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-09-08 17:16:26 +0200] information/Application: Reload done, old process shutting down. Child process with PID '25681' is taking over.
[2022-09-08 17:16:26 +0200] information/Application: Received request to shut down.
[2022-09-08 17:16:27 +0200] information/Application: Shutting down...
[2022-09-08 17:16:27 +0200] information/CheckerComponent: 'checker' stopped.
[2022-09-08 17:16:27 +0200] information/NotificationComponent: 'notification' stopped.
[2022-09-08 17:16:27 +0200] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2022-09-08 17:16:27 +0200] information/IcingaApplication: Icinga has shut down.
[2022-09-08 17:16:27 +0200] information/ConfigObject: Restoring program state from file '/var/lib/icinga2/icinga2.state'
[2022-09-08 17:16:27 +0200] information/ConfigObject: Restored 293 objects. Loaded 0 new objects without state.
[2022-09-08 17:16:27 +0200] information/ConfigItem: Triggering Start signal for config items
[2022-09-08 17:16:27 +0200] information/FileLogger: 'debug-file' started.
[2022-09-08 17:16:27 +0200] information/FileLogger: 'main-log' started.
[2022-09-08 17:16:27 +0200] information/NotificationComponent: 'notification' started.
[2022-09-08 17:16:27 +0200] information/CheckerComponent: 'checker' started.
[2022-09-08 17:16:27 +0200] information/ConfigItem: Activated all objects.
[2022-09-08 17:16:31 +0200] information/Application: Got reload command: Starting new instance.
[2022-09-08 17:16:31 +0200] information/cli: Loading configuration file(s).
[2022-09-08 17:16:31 +0200] information/ConfigItem: Committing config item(s).
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 12 Notifications.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 3 ServiceGroups.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 3 Zones.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 11 Services.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 1 User.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 1 Host.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 1 Downtime.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 3 HostGroups.
[2022-09-08 17:16:31 +0200] information/ConfigItem: Instantiated 244 CheckCommands.
[2022-09-08 17:16:31 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-09-08 17:16:31 +0200] information/Application: Reload done, old process shutting down. Child process with PID '46511' is taking over.
[2022-09-08 17:16:31 +0200] information/Application: Received request to shut down.
[2022-09-08 17:16:32 +0200] information/Application: Shutting down...
[2022-09-08 17:16:32 +0200] information/CheckerComponent: 'checker' stopped.
[2022-09-08 17:16:32 +0200] information/NotificationComponent: 'notification' stopped.
[2022-09-08 17:16:32 +0200] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2022-09-08 17:16:32 +0200] information/IcingaApplication: Icinga has shut down.
[2022-09-08 17:16:32 +0200] information/ConfigObject: Restoring program state from file '/var/lib/icinga2/icinga2.state'
[2022-09-08 17:16:32 +0200] information/ConfigObject: Restored 293 objects. Loaded 0 new objects without state.
[2022-09-08 17:16:32 +0200] information/ConfigItem: Triggering Start signal for config items
[2022-09-08 17:16:32 +0200] information/FileLogger: 'debug-file' started.
[2022-09-08 17:16:32 +0200] information/FileLogger: 'main-log' started.
[2022-09-08 17:16:32 +0200] information/NotificationComponent: 'notification' started.
[2022-09-08 17:16:32 +0200] information/CheckerComponent: 'checker' started.
[2022-09-08 17:16:32 +0200] information/ConfigItem: Activated all objects.
[2022-09-08 17:16:37 +0200] information/Application: Got reload command: Starting new instance.
[2022-09-08 17:16:37 +0200] information/cli: Loading configuration file(s).
[2022-09-08 17:16:37 +0200] information/ConfigItem: Committing config item(s).
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 12 Notifications.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 3 ServiceGroups.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 3 Zones.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 11 Services.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 1 User.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 1 Host.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 1 Downtime.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 3 HostGroups.
[2022-09-08 17:16:37 +0200] information/ConfigItem: Instantiated 244 CheckCommands.
[2022-09-08 17:16:37 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-09-08 17:16:37 +0200] information/Application: Reload done, old process shutting down. Child process with PID '53380' is taking over.
[2022-09-08 17:16:37 +0200] information/Application: Received request to shut down.
[2022-09-08 17:16:40 +0200] information/Application: Shutting down...
[2022-09-08 17:16:40 +0200] information/CheckerComponent: 'checker' stopped.
[2022-09-08 17:16:40 +0200] information/NotificationComponent: 'notification' stopped.
[2022-09-08 17:16:40 +0200] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2022-09-08 17:16:40 +0200] information/IcingaApplication: Icinga has shut down.
[2022-09-08 17:16:40 +0200] information/ConfigObject: Restoring program state from file '/var/lib/icinga2/icinga2.state'
[2022-09-08 17:16:40 +0200] information/ConfigObject: Restored 293 objects. Loaded 0 new objects without state.
[2022-09-08 17:16:40 +0200] information/ConfigItem: Triggering Start signal for config items
[2022-09-08 17:16:40 +0200] information/FileLogger: 'debug-file' started.
[2022-09-08 17:16:40 +0200] information/FileLogger: 'main-log' started.
[2022-09-08 17:16:40 +0200] information/NotificationComponent: 'notification' started.
[2022-09-08 17:16:40 +0200] information/CheckerComponent: 'checker' started.
[2022-09-08 17:16:40 +0200] information/ConfigItem: Activated all objects.

Awesome, just awesome!

[sthen]

Ah I think this explains something I had noticed occasionally with director but hadn't figured out how to reproduce reliably. I can't say 100% for sure, but I think this does fix that.

btw you can grab a patch from a github PR with this construct (then you need "make makesum" to update distinfo):

MASTER_SITES0 = https://github.com/Icinga/icinga2/pull/
PATCHFILES +=   9518.patch:0
PATCH_DIST_STRIP = -p1

[Al2Klimov]

Apropos patch from a github PR: Feel free to include all this stuff as patches into ports as long as not merged here.