icingadb icon indicating copy to clipboard operation
icingadb copied to clipboard
verified publisher icon Icinga

Crash when transaction can't complete

Open deric opened this issue 1 year ago • 2 comments

Transaction failure results in service crash

pq: could not serialize access due to read/write dependencies among transactions
                                                         can't perform "INSERT INTO \"icingadb_instance\" (\"icinga2_performance_data_enabled\", \"responsible\", \"icinga2_version\", \"icinga2_event_handlers_enabled\", \"endpoint_id\", \"heartbeat\", \"id\", \"icinga2_start_time\", \"icinga2_flap_detection_enabled\", \"icinga2_active_host_checks_enabled\", \"environment_id\", \"icinga2_active_service_checks_enabled\", \"icinga2_notifications_enabled\") VALUES (:icinga2_performance_data_enabled,:responsible,:icinga2_version,:icinga2_event_handlers_enabled,:endpoint_id,:heartbeat,:id,:icinga2_start_time,:icinga2_flap_detection_enabled,:icinga2_active_host_checks_enabled,:environment_id,:icinga2_active_service_checks_enabled,:icinga2_notifications_enabled) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \"icinga2_performance_data_enabled\" = EXCLUDED.\"icinga2_performance_data_enabled\",\"responsible\" = EXCLUDED.\"responsible\",\"icinga2_version\" = EXCLUDED.\"icinga2_version\",\"icinga2_event_handlers_enabled\" = EXCLUDED.\"icinga2_event_handlers_enabled\",\"endpoint_id\" = EXCLUDED.\"endpoint_id\",\"heartbeat\" = EXCLUDED.\"heartbeat\",\"id\" = EXCLUDED.\"id\",\"icinga2_start_time\" = EXCLUDED.\"icinga2_start_time\",\"icinga2_flap_detection_enabled\" = EXCLUDED.\"icinga2_flap_detection_enabled\",\"icinga2_active_host_checks_enabled\" = EXCLUDED.\"icinga2_active_host_checks_enabled\",\"environment_id\" = EXCLUDED.\"environment_id\",\"icinga2_active_service_checks_enabled\" = EXCLUDED.\"icinga2_active_service_checks_enabled\",\"icinga2_notifications_enabled\" = EXCLUDED.\"icinga2_notifications_enabled\""
                                                         github.com/icinga/icingadb/internal.CantPerformQuery
                                                                 github.com/icinga/icingadb/internal/internal.go:30
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).realize.func1
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:368
                                                         github.com/icinga/icingadb/pkg/retry.WithBackoff
                                                                 github.com/icinga/icingadb/pkg/retry/retry.go:60
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).realize
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:284
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:221
                                                         runtime.goexit
                                                                 runtime/asm_amd64.s:1695
                                                         can't perform "INSERT INTO \"icingadb_instance\" (\"icinga2_flap_detection_enabled\", \"icinga2_active_host_checks_enabled\", \"environment_id\", \"icinga2_active_service_checks_enabled\", \"icinga2_notifications_enabled\", \"icinga2_event_handlers_enabled\", \"icinga2_performance_data_enabled\", \"responsible\", \"icinga2_version\", \"endpoint_id\", \"heartbeat\", \"id\", \"icinga2_start_time\") VALUES (:icinga2_flap_detection_enabled,:icinga2_active_host_checks_enabled,:environment_id,:icinga2_active_service_checks_enabled,:icinga2_notifications_enabled,:icinga2_event_handlers_enabled,:icinga2_performance_data_enabled,:responsible,:icinga2_version,:endpoint_id,:heartbeat,:id,:icinga2_start_time) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \"icinga2_flap_detection_enabled\" = EXCLUDED.\"icinga2_flap_detection_enabled\",\"icinga2_active_host_checks_enabled\" = EXCLUDED.\"icinga2_active_host_checks_enabled\",\"environment_id\" = EXCLUDED.\"environment_id\",\"icinga2_active_service_checks_enabled\" = EXCLUDED.\"icinga2_active_service_checks_enabled\",\"icinga2_notifications_enabled\" = EXCLUDED.\"icinga2_notifications_enabled\",\"icinga2_event_handlers_enabled\" = EXCLUDED.\"icinga2_event_handlers_enabled\",\"icinga2_performance_data_enabled\" = EXCLUDED.\"icinga2_performance_data_enabled\",\"responsible\" = EXCLUDED.\"responsible\",\"icinga2_version\" = EXCLUDED.\"icinga2_version\",\"endpoint_id\" = EXCLUDED.\"endpoint_id\",\"heartbeat\" = EXCLUDED.\"heartbeat\",\"id\" = EXCLUDED.\"id\",\"icinga2_start_time\" = EXCLUDED.\"icinga2_start_time\"": pq: could not serialize access due to read/write dependencies among transactions
                                                         github.com/icinga/icingadb/pkg/retry.WithBackoff
                                                                 github.com/icinga/icingadb/pkg/retry/retry.go:75
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).realize
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:284
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:221
                                                         runtime.goexit
                                                                 runtime/asm_amd64.s:1695
                                                         HA aborted
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).abort.func1
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:131
                                                         sync.(*Once).doSlow
                                                                 sync/once.go:74
                                                         sync.(*Once).Do
                                                                 sync/once.go:65
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).abort
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:129
                                                         github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                                                 github.com/icinga/icingadb/pkg/icingadb/ha.go:233
                                                         runtime.goexit
                                                                 runtime/asm_amd64.s:1695
                                                         HA exited with an error
                                                         main.run
                                                                 github.com/icinga/icingadb/cmd/icingadb/main.go:335
                                                         main.main
                                                                 github.com/icinga/icingadb/cmd/icingadb/main.go:37
                                                         runtime.main
                                                                 runtime/proc.go:271
                                                         runtime.goexit
                                                                 runtime/asm_amd64.s:1695
icingadb.service: Main process exited, code=exited, status=1/FAILURE
icingadb.service: Failed with result 'exit-code'.
  • Icinga DB version: v1.2.0
  • Icinga 2 version: 2.14.2
  • Operating System and version: Debian 12

deric avatar Sep 12 '24 17:09 deric

Thanks for your bug report. Such an issue should be retried, as being addressed in Icinga/icinga-go-library#59. However, based on your stack trace, I am unsure if it wasn't even retried. Could you please supply more logs, best at least the last 30 minutes prior to the crash?

oxzi avatar Sep 13 '24 09:09 oxzi

Sure, the server was overloaded I have upgrade it. However icingadb shouldn't be crashing...

Thu 2024-09-12 17:03:24 UTC icingadb[1121651]: runtime-updates: Upserted 10 ServiceState items
Thu 2024-09-12 17:03:39 UTC icingadb[1121651]: overdue-sync: Synced 44 service overdue indicators
Thu 2024-09-12 17:03:59 UTC icingadb[1121651]: overdue-sync: Synced 19 service overdue indicators
Thu 2024-09-12 17:04:08 UTC icingadb[1121651]: heartbeat: Lost Icinga heartbeat
Thu 2024-09-12 17:04:08 UTC icingadb[1121651]: high-availability: Lost heartbeat
Thu 2024-09-12 17:04:08 UTC icingadb[1121651]: Handing over
Thu 2024-09-12 17:05:08 UTC icingadb[1121651]: heartbeat: Waiting for Icinga heartbeat
Thu 2024-09-12 17:05:38 UTC icingadb[1121651]: heartbeat: Received Icinga heartbeat
Thu 2024-09-12 17:05:38 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:05:38 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:05:39 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:05:41 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:05:41 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:44 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:05:44 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:46 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:48 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:51 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:53 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:06:53 UTC icingadb[1121651]: high-availability: Instance updated/inserted successfully after error
Thu 2024-09-12 17:06:57 UTC icingadb[1121651]: history-sync: Synced 1 state history items
Thu 2024-09-12 17:08:05 UTC icingadb[1121651]: pq: could not serialize access due to read/write dependencies among transactions
                                               can't perform "INSERT INTO \"icingadb_instance\" (\"icinga2_performance_data_enabled\", \"responsible\", \"icinga2_version\", \"icinga2_event_hand>
                                               github.com/icinga/icingadb/internal.CantPerformQuery
                                                       github.com/icinga/icingadb/internal/internal.go:30
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).realize.func1
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:368
                                               github.com/icinga/icingadb/pkg/retry.WithBackoff
                                                       github.com/icinga/icingadb/pkg/retry/retry.go:60
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).realize
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:284
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:221
                                               runtime.goexit
                                                       runtime/asm_amd64.s:1695
                                               can't perform "INSERT INTO \"icingadb_instance\" (\"icinga2_flap_detection_enabled\", \"icinga2_active_host_checks_enabled\", \"environment_id\", >
                                               github.com/icinga/icingadb/pkg/retry.WithBackoff
                                                       github.com/icinga/icingadb/pkg/retry/retry.go:75
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).realize
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:284
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:221
                                               runtime.goexit
                                                       runtime/asm_amd64.s:1695
                                               HA aborted
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).abort.func1
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:131
                                               sync.(*Once).doSlow
                                                       sync/once.go:74
                                               sync.(*Once).Do
                                                       sync/once.go:65
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).abort
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:129
                                               github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                                       github.com/icinga/icingadb/pkg/icingadb/ha.go:233
                                               runtime.goexit
                                                       runtime/asm_amd64.s:1695
                                               HA exited with an error
                                               main.run
                                                       github.com/icinga/icingadb/cmd/icingadb/main.go:335
                                               main.main
                                                       github.com/icinga/icingadb/cmd/icingadb/main.go:37
                                               runtime.main
                                                       runtime/proc.go:271
                                               runtime.goexit
                                                       runtime/asm_amd64.s:1695
Thu 2024-09-12 17:08:05 UTC systemd[1]: icingadb.service: Main process exited, code=exited, status=1/FAILURE
Thu 2024-09-12 17:08:05 UTC systemd[1]: icingadb.service: Failed with result 'exit-code'.
Thu 2024-09-12 17:08:05 UTC systemd[1]: icingadb.service: Consumed 3min 35.643s CPU time.
Thu 2024-09-12 17:24:38 UTC systemd[1]: Starting icingadb.service - Icinga DB...
Thu 2024-09-12 17:24:38 UTC icingadb[1184381]: Starting Icinga DB daemon (v1.2.0)
Thu 2024-09-12 17:24:38 UTC systemd[1]: Started icingadb.service - Icinga DB.
Thu 2024-09-12 17:24:38 UTC icingadb[1184381]: Connecting to database at 'icm.de.recombee.net:0'
Thu 2024-09-12 17:24:38 UTC icingadb[1184381]: Connecting to Redis at 'localhost:6380'
Thu 2024-09-12 17:24:38 UTC icingadb[1184381]: Starting history sync
Thu 2024-09-12 17:24:39 UTC icingadb[1184381]: heartbeat: Received Icinga heartbeat
Thu 2024-09-12 17:24:39 UTC icingadb[1184381]: high-availability: Another instance is active
Thu 2024-09-12 17:24:58 UTC icingadb[1184381]: history-sync: Synced 31 notification history items
Thu 2024-09-12 17:24:58 UTC icingadb[1184381]: history-sync: Synced 2745 state history items
Thu 2024-09-12 17:25:18 UTC icingadb[1184381]: history-sync: Synced 24 state history items
Thu 2024-09-12 17:25:38 UTC icingadb[1184381]: history-sync: Synced 2 notification history items
Thu 2024-09-12 17:25:38 UTC icingadb[1184381]: history-sync: Synced 79 state history items
Thu 2024-09-12 17:25:58 UTC icingadb[1184381]: history-sync: Synced 18 state history items
Thu 2024-09-12 17:26:18 UTC icingadb[1184381]: history-sync: Synced 80 state history items
Thu 2024-09-12 17:26:18 UTC icingadb[1184381]: history-sync: Synced 1 notification history items
Thu 2024-09-12 17:26:38 UTC icingadb[1184381]: history-sync: Synced 61 state history items
Thu 2024-09-12 17:26:38 UTC icingadb[1184381]: history-sync: Synced 2 notification history items
Thu 2024-09-12 17:26:58 UTC icingadb[1184381]: history-sync: Synced 47 state history items
Thu 2024-09-12 17:27:18 UTC icingadb[1184381]: history-sync: Synced 1 notification history items
Thu 2024-09-12 17:27:18 UTC icingadb[1184381]: history-sync: Synced 25 state history items
Thu 2024-09-12 17:27:38 UTC icingadb[1184381]: history-sync: Synced 36 state history items
Thu 2024-09-12 17:27:58 UTC icingadb[1184381]: history-sync: Synced 4 notification history items
Thu 2024-09-12 17:27:58 UTC icingadb[1184381]: history-sync: Synced 37 state history items
Thu 2024-09-12 17:28:18 UTC icingadb[1184381]: history-sync: Synced 34 state history items
Thu 2024-09-12 17:28:38 UTC icingadb[1184381]: history-sync: Synced 1 notification history items
Thu 2024-09-12 17:28:38 UTC icingadb[1184381]: history-sync: Synced 51 state history items
Thu 2024-09-12 17:28:58 UTC icingadb[1184381]: history-sync: Synced 53 state history items
Thu 2024-09-12 17:29:18 UTC icingadb[1184381]: history-sync: Synced 62 state history items
Thu 2024-09-12 17:29:38 UTC icingadb[1184381]: high-availability: Another instance is active
Thu 2024-09-12 17:29:38 UTC icingadb[1184381]: history-sync: Synced 78 state history items
Thu 2024-09-12 17:29:58 UTC icingadb[1184381]: history-sync: Synced 56 state history items
Thu 2024-09-12 17:30:10 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:30:11 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:30:14 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:30:15 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:30:15 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:30:17 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:30:18 UTC icingadb[1184381]: history-sync: Synced 2 notification history items
Thu 2024-09-12 17:30:18 UTC icingadb[1184381]: history-sync: Synced 27 state history items
Thu 2024-09-12 17:30:18 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:30:19 UTC icingadb[1184381]: database: Can't connect to database. Retrying
Thu 2024-09-12 17:05:44 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:05:44 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:46 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:48 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:51 UTC icingadb[1121651]: high-availability: Can't update or insert instance. Retrying
Thu 2024-09-12 17:05:53 UTC icingadb[1121651]: high-availability: Another instance is active
Thu 2024-09-12 17:06:53 UTC icingadb[1121651]: high-availability: Instance updated/inserted successfully after error
Thu 2024-09-12 17:06:57 UTC icingadb[1121651]: history-sync: Synced 1 state history items

deric avatar Sep 13 '24 10:09 deric

Should be fixed by https://github.com/Icinga/icinga-go-library/pull/59

yhabteab avatar Oct 18 '24 10:10 yhabteab