icinga2 icon indicating copy to clipboard operation
icinga2 copied to clipboard

Published 20 hours ago • verified publisher icon Icinga

Dependency not breaking as expected

Open matthenning opened this issue 2 years ago • 7 comments

Describe the bug

Seemingly after updating to the latest Icinga 2 version this dependency started behaving in unexpected ways. This dependency should suppress all checks and notifications when the cluster zone is disconnected. Please note that I'm not 100% sure this is related to the latest update but the first false alert occurred last night after applying the patches yesterday.

Maybe I'm missing something obvious here.

I have observed two scenarios: Scenario 1: The dependency breaks, but after the parent service is on 3 to 4 of 10 retries, so still in soft state, the dependency no longer breaks and the child checks start checking and sending notifications. Note that the cluster zone check is critical and the service checks below it are overdue, that is what I expect to happen: image

image

Now a few moments later the cluster zone is still critical, but the other checks are starting to check again: image

image

Scenario 2: Identical to scenario 1, but the dependency never breaks in the first place.

Weirdly enough the debug log suggests that everything is working correctly. Debuglog, normal state with cluster zone connected:

[2022-12-07 13:10:23 +0100] notice/Dependency: Dependency 'redacted-server-1!App - Icinga 2 Service!Disable Agent Checks if Cluster Zone NOT-OK' failed: Parent service 'redacted-server-1!App - Icinga 2 Cluster Zone' is in a soft state.
[2022-12-07 13:10:23 +0100] notice/Dependency: Dependency 'redacted-server-1!App - Icinga 2 Service!Disable Agent Checks if Cluster Zone NOT-OK' passed: Parent service 'redacted-server-1!App - Icinga 2 Cluster Zone' matches state filter.

Debulog, failed state with cluster zone disconnected. As expected the dependency fails:

[2022-12-07 13:09:17 +0100] notice/Dependency: Dependency 'redacted-server-1!App - Icinga 2 Service!Disable Agent Checks if Cluster Zone NOT-OK' failed: Parent service 'redacted-server-1!App - Icinga 2 Cluster Zone' is in a soft state.
[2022-12-07 13:09:17 +0100] notice/Dependency: Dependency 'redacted-server-1!App - Icinga 2 Service!Disable Agent Checks if Cluster Zone NOT-OK' failed. Parent service 'redacted-server-1!App - Icinga 2 Cluster Zone' is CRITICAL

The dependency definition:

Object 'redacted-server-01!App - Icinga 2 Service!Disable Agent Checks if Cluster Zone NOT-OK' of type 'Dependency':
  % declared in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_apply.conf', lines 43:1-43:73
  * __name = "redacted-server-01!App - Icinga 2 Service!Disable Agent Checks if Cluster Zone NOT-OK"
  * child_host_name = "redacted-server-01"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_apply.conf', lines 43:1-43:73
  * child_service_name = "App - Icinga 2 Service"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_apply.conf', lines 43:1-43:73
  * disable_checks = true
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_templates.conf', lines 2:5-2:25
  * disable_notifications = true
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_templates.conf', lines 3:5-3:32
  * ignore_soft_states = false
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_templates.conf', lines 4:5-4:29
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_apply.conf', lines 46:5-46:30
  * name = "Disable Agent Checks if Cluster Zone NOT-OK"
  * package = "director"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_apply.conf', lines 43:1-43:73
  * parent_host_name = "redacted-server-01"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_apply.conf', lines 43:1-43:73
  * parent_service_name = "App - Icinga 2 Cluster Zone"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_apply.conf', lines 48:5-48:55
  * period = ""
  * source_location
    * first_column = 1
    * first_line = 43
    * last_column = 73
    * last_line = 43
    * path = "/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_apply.conf"
  * states = [ "OK" ]
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_templates.conf', lines 5:5-5:19
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_apply.conf', lines 49:5-49:19
  * templates = [ "Disable Agent Checks if Cluster Zone NOT-OK", "Generic Dependency" ]
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_apply.conf', lines 43:1-43:73
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_templates.conf', lines 1:0-1:39
  * type = "Dependency"
  * vars = null
  * zone = "master"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_apply.conf', lines 43:1-43:73

Child service definition:

Object 'redacted-server-01!App - Icinga 2 Service' of type 'Service':
  % declared in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf', lines 348:1-348:38
  * __name = "redacted-server-01!App - Icinga 2 Service"
  * action_url = ""
  * check_command = "service-windows"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 506:5-506:37
  * check_interval = 60
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 5:5-5:23
  * check_period = ""
  * check_timeout = 30
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 7:5-7:23
  * command_endpoint = "redacted-server-01"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 507:5-507:32
  * display_name = "App - Icinga 2 Service"
  * enable_active_checks = true
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 9:5-9:31
  * enable_event_handler = true
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 11:5-11:31
  * enable_flapping = true
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 12:5-12:26
  * enable_notifications = true
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 8:5-8:31
  * enable_passive_checks = true
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 10:5-10:32
  * enable_perfdata = true
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 13:5-13:26
  * event_command = ""
  * flapping_ignore_states = null
  * flapping_threshold = 0
  * flapping_threshold_high = 30
  * flapping_threshold_low = 25
  * groups = [ ]
  * host_name = "redacted-server-01"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf', lines 348:1-348:38
  * icon_image = "win.png"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 39:5-39:26
  * icon_image_alt = ""
  * max_check_attempts = 4
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 4:5-4:28
  * name = "App - Icinga 2 Service"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf', lines 348:1-348:38
  * notes = ""
  * notes_url = ""
  * package = "director"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf', lines 348:1-348:38
  * retry_interval = 30
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 6:5-6:24
  * source_location
    * first_column = 1
    * first_line = 348
    * last_column = 38
    * last_line = 348
    * path = "/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf"
  * templates = [ "App - Icinga 2 Service", "AGENT Windows Icinga 2", "Windows", "System Metrics", "00_generic_service", "host var overrides (Director)" ]
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf', lines 348:1-348:38
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 503:1-503:41
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 36:1-36:26
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 1:0-1:32
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 884:1-884:37
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/001-director-basics.conf', lines 28:3-28:43
  * type = "Service"
  * vars
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/001-director-basics.conf', lines 41:7-41:51
    * monitoring_grafana_dashboard_id = "VMGItsiik"
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 885:5-885:54
    * monitoring_grafana_panel_id = 2
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 886:5-886:40
    * monitoring_no_graph = true
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 508:5-508:35
    * notification_email_begin = 10
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 887:5-887:38
    * notification_service_states = [ "Critical", "Warning" ]
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 888:5-888:64
    * notifications_enabled = true
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 889:5-889:37
    * service_win_service = "icinga2"
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 509:5-509:40
  * volatile = false
  * zone = "master"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf', lines 348:1-348:38

Parent service definition:

Object 'redacted-server-01!App - Icinga 2 Cluster Zone' of type 'Service':
  % declared in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf', lines 295:1-295:43
  * __name = "redacted-server-01!App - Icinga 2 Cluster Zone"
  * action_url = ""
  * check_command = "cluster-zone"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 88:5-88:34
  * check_interval = 15
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 20:5-20:23
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 90:5-90:24
  * check_period = ""
  * check_timeout = 30
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 22:5-22:23
  * command_endpoint = ""
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 92:5-92:27
  * display_name = "App - Icinga 2 Cluster Zone"
  * enable_active_checks = true
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 24:5-24:31
  * enable_event_handler = true
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 26:5-26:31
  * enable_flapping = false
  * enable_notifications = true
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 23:5-23:31
  * enable_passive_checks = true
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 25:5-25:32
  * enable_perfdata = true
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 27:5-27:26
  * event_command = ""
  * flapping_ignore_states = null
  * flapping_threshold = 0
  * flapping_threshold_high = 30
  * flapping_threshold_low = 25
  * groups = [ ]
  * host_name = "redacted-server-01"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf', lines 295:1-295:43
  * icon_image = "icinga2.png"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 65:5-65:30
  * icon_image_alt = ""
  * max_check_attempts = 10
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 19:5-19:28
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 89:5-89:29
  * name = "App - Icinga 2 Cluster Zone"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf', lines 295:1-295:43
  * notes = ""
  * notes_url = ""
  * package = "director"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf', lines 295:1-295:43
  * retry_interval = 60
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 21:5-21:24
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 91:5-91:23
  * source_location
    * first_column = 1
    * first_line = 295
    * last_column = 43
    * last_line = 295
    * path = "/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf"
  * templates = [ "App - Icinga 2 Cluster Zone", "REMOTE Icinga 2 Cluster Zone", "Icinga 2", "Application Metrics", "00_generic_service", "host var overrides (Director)" ]
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf', lines 295:1-295:43
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 85:1-85:47
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 62:1-62:27
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 16:1-16:38
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 884:1-884:37
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/001-director-basics.conf', lines 28:3-28:43
  * type = "Service"
  * vars
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/001-director-basics.conf', lines 41:7-41:51
    * cluster_zone = "$host.vars.app_icinga2_cluster_zone$"
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 93:5-93:62
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf', lines 299:5-299:62
    * monitoring_grafana_dashboard_id = "VMGItsiik"
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 885:5-885:54
    * monitoring_grafana_panel_id = 2
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 886:5-886:40
    * notification_email_begin = 10
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 887:5-887:38
    * notification_methods = [ "mail", "sms" ]
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 94:5-94:49
    * notification_service_states = [ "Critical", "Warning" ]
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 888:5-888:64
    * notification_timeperiod = "24x7"
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 95:5-95:41
    * notifications_enabled = true
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 889:5-889:37
      % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_templates.conf', lines 96:5-96:37
  * volatile = false
  * zone = "master"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf', lines 295:1-295:43

To Reproduce

I'm not sure which part of the config causes the issue so it's hard for me to provide details about reproducing the issue. For now I can reliably reproduce the issue on our environment.

Expected behavior

The dependency stays broken and the child service don't start checking or sending notifications.

Your Environment

Include as many relevant details about the environment you experienced the problem in

  • Version used (icinga2 --version): r2.13.6-1
  • Operating System and version: Red Hat Enterprise Linux Server release 7.9 (Maipo)
  • Enabled features (icinga2 feature list): api checker debuglog icingadb ido-mysql influxdb mainlog notification
  • Icinga Web 2 version and modules (System - About): image
  • Config validation (icinga2 daemon -C):
[2022-12-07 13:36:21 +0100] information/cli: Icinga application loader (version: r2.13.6-1)
[2022-12-07 13:36:21 +0100] information/cli: Loading configuration file(s).
[2022-12-07 13:36:21 +0100] information/ConfigItem: Committing config item(s).
[2022-12-07 13:36:21 +0100] information/ApiListener: My API identity: redacted-master-1
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'Service SMS Notification WITHWARN' (in /etc/icinga2/zones.d/master/notifications.conf: 131:1-131:65) for type 'Notification' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'Service Jira Notification' (in /etc/icinga2/zones.d/master/notifications.conf: 147:1-147:57) for type 'Notification' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'Host Jira Notification' (in /etc/icinga2/zones.d/master/notifications.conf: 83:1-83:51) for type 'Notification' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'Maintenance Daily 0300 to 0600' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/master/scheduled_downtime_apply.conf: 21:1-21:64) for type 'ScheduledDowntime' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'Maintenance Saturday 0800 to 1300' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/master/scheduled_downtime_apply.conf: 85:1-85:67) for type 'ScheduledDowntime' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'Maintenance Daily 0300 to 0600' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/master/scheduled_downtime_apply.conf: 37:1-37:67) for type 'ScheduledDowntime' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'Maintenance Saturday 0800 to 1300' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/master/scheduled_downtime_apply.conf: 95:1-95:70) for type 'ScheduledDowntime' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'Maintenance Saturday 0800 to 1500 (Services only)' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/master/scheduled_downtime_apply.conf: 291:1-291:86) for type 'ScheduledDowntime' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - MySQL Health' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 215:1-215:34) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - Logstash Health' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 231:1-231:37) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - NGINX Process' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 239:1-239:35) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - Icingaweb 2 Status' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 247:1-247:40) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - Grafana Status' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 255:1-255:36) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - Kibana Status' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 263:1-263:35) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - MSSQL Service ReportServer' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 399:1-399:48) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'System - Disk L:' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 438:1-438:32) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'System - Disk ' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 563:1-563:74) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - IQ Process' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 695:1-695:32) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'System - Disk S:' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 952:1-952:32) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'System - Disk B:' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 969:1-969:32) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - IQ DB Space Usage' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 987:1-987:39) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - MSSQL Backup: Last Restore - ' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1250:1-1250:105) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'System - Disk W:' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1327:1-1327:32) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'System - Disk V:' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1336:1-1336:32) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - Verint Service SQL Server Agent' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1461:1-1461:53) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'System - Disk H:' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1693:1-1693:32) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - Automic Service Manager' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1702:1-1702:45) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - FINDIG Apache2' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1763:1-1763:36) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - FINDIG bpcs.config' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1771:1-1771:40) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - FINDIG bpcs.main' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1779:1-1779:38) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - FINDIG pts' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1787:1-1787:32) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - RedGate Base Monitor' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1815:1-1815:42) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - RedGate Web Service' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1824:1-1824:41) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - Automic Execution ' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1843:1-1843:94) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - MSSQL Meta DB Login' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1969:1-1969:41) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'System - Disk R:' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 1999:1-1999:32) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'App - RabbitMQ Queues' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/service_apply.conf: 2067:1-2067:37) for type 'Service' does not match anywhere!
[2022-12-07 13:36:23 +0100] warning/ApplyRule: Apply rule 'Disable MSSQL Checks if Login NOT-OK' (in /var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/director-global/dependency_apply.conf: 31:1-31:66) for type 'Dependency' does not match anywhere!
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 1 InfluxdbWriter.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 15 UserGroups.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 15 TimePeriods.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 10 Users.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 4033 Services.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 4713 ScheduledDowntimes.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 139 Zones.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 1 NotificationCommand.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 5986 Notifications.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 145 Hosts.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 27 HostGroups.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 8773 Dependencies.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 67 Comments.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 4736 Downtimes.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 130 Endpoints.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 2 FileLoggers.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 6 ApiUsers.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 324 CheckCommands.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2022-12-07 13:36:23 +0100] information/ConfigItem: Instantiated 1 IcingaDB.
[2022-12-07 13:36:24 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-12-07 13:36:24 +0100] information/cli: Finished validating the configuration file(s).
  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

Master nodes:

Object 'redacted-master-1' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 6:1-6:57
  * __name = "redacted-master-1"
  * host = ""
  * log_duration = 86400
  * name = "redacted-master-1"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 57
    * last_line = 6
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "redacted-master-1" ]
    % = modified in '/etc/icinga2/zones.conf', lines 6:1-6:57
  * type = "Endpoint"
  * zone = ""

Object 'redacted-master-2' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 9:1-9:57
  * __name = "redacted-master-2"
  * host = "redacted-master-2"
    % = modified in '/etc/icinga2/zones.conf', lines 10:2-10:49
  * log_duration = 86400
  * name = "redacted-master-2"
  * package = "_etc"
  * port = "5665"
    % = modified in '/etc/icinga2/zones.conf', lines 11:2-11:12
  * source_location
    * first_column = 1
    * first_line = 9
    * last_column = 57
    * last_line = 9
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "redacted-master-2" ]
    % = modified in '/etc/icinga2/zones.conf', lines 9:1-9:57
  * type = "Endpoint"
  * zone = ""



Object 'master' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 14:1-14:20
  * __name = "master"
  * endpoints = [ "redacted-master-1", "redacted-master-2" ]
    % = modified in '/etc/icinga2/zones.conf', lines 15:2-15:101
  * global = false
  * name = "master"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 14
    * last_column = 20
    * last_line = 14
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "master" ]
    % = modified in '/etc/icinga2/zones.conf', lines 14:1-14:20
  * type = "Zone"
  * zone = ""

Affected server:

Object 'redacted-server-01' of type 'Zone':
  % declared in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/master/zones.conf', lines 145:1-145:42
  * __name = "redacted-server-01"
  * endpoints = [ "redacted-server-01" ]
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/master/zones.conf', lines 147:5-147:50
  * global = false
  * name = "redacted-server-01"
  * package = "director"
  * parent = "master"
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/master/zones.conf', lines 146:5-146:21
  * source_location
    * first_column = 1
    * first_line = 145
    * last_column = 42
    * last_line = 145
    * path = "/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/master/zones.conf"
  * templates = [ "redacted-server-01" ]
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/master/zones.conf', lines 145:1-145:42
  * type = "Zone"
  * zone = "master"


Object 'redacted-server-01' of type 'Endpoint':
  % declared in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/master/endpoints.conf', lines 76:1-76:46
  * __name = "redacted-server-01"
  * host = ""
  * log_duration = 86400
  * name = "redacted-server-01"
  * package = "director"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 76
    * last_column = 46
    * last_line = 76
    * path = "/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/master/endpoints.conf"
  * templates = [ "redacted-server-01" ]
    % = modified in '/var/lib/icinga2/api/packages/director/2d9419f0-d36c-450b-a9d0-fd26312e1e44/zones.d/master/endpoints.conf', lines 76:1-76:46
  * type = "Endpoint"
  * zone = "master"

Please let me know if you need any additional information.

matthenning avatar Dec 07 '22 12:12 matthenning

Added missing zone definitions in OP.

matthenning avatar Dec 07 '22 12:12 matthenning

Hello Matthias!

from which Icinga version did you upgrade?

Al2Klimov avatar Feb 07 '23 15:02 Al2Klimov

The latest update of the icinga2 package was installed 2022-12-06 and before that on 2022-08-31 so I guess I updated from 2.13.5, as that version was released earlier in August. We're usually at most one month behind on our master servers.

matthenning avatar Feb 08 '23 10:02 matthenning

How many dependencies the service depends on?

Al2Klimov avatar May 09 '23 08:05 Al2Klimov

In this case the service is child of two dependencies, if my grep didn't fail me. The on mentioned in OP and a second one which I think was created to disable all Service checks as soon as the Host is in a soft not-OK state:

# icinga2 object list --type Dependency | grep 'child_host_name = "redacted-server-01' -A 2 -B 3 | grep 'child_service_name = "App - Icinga 2 Service"' -B 3

  * __name = "redacted-server-01!App - Icinga 2 Service!MANUAL Disable Agent Checks if Cluster Zone NOT-OK"
  * child_host_name = "redacted-server-01"
    % = modified in '/etc/icinga2/zones.d/global-templates/dependencies.conf', lines 8:1-8:80
  * child_service_name = "App - Icinga 2 Service"
--
  * __name = "redacted-server-01!App - Icinga 2 Service!Disable Service Checks if Host NOT-OK"
  * child_host_name = "redacted-server-01"
    % = modified in '/etc/icinga2/zones.d/global-templates/dependencies.conf', lines 1:0-1:73
  * child_service_name = "App - Icinga 2 Service"

The second one's definition:

apply Dependency "MANUAL Disable Service Checks if Host NOT-OK" to Service {
    disable_checks = true
    disable_notifications = true
    ignore_soft_states = false
    assign where true
}

matthenning avatar May 09 '23 11:05 matthenning

This should help you:

  • #8218

Please test v2.14.x once released.

Al2Klimov avatar May 09 '23 11:05 Al2Klimov

This sounds perfect, thank you.

matthenning avatar May 09 '23 11:05 matthenning