icingadb icon indicating copy to clipboard operation
icingadb copied to clipboard
verified publisher icon Icinga

Competing HA takeover results in both instances becoming active

Open yhabteab opened this issue 1 year ago • 1 comments

Describe the bug

Apparently there is a race condition in our HA realize handlings that can occur from time to time, causing both Icinga DB instances to become active/responsible at the same time. It is possible for the following to happen if both instances access the icingadb_instance table at the same time.

Since both instances are trying to update that table quite often[^1], it is to be expected that they will deadlock each other, as can be seen in the logs.

{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 11:57:43",
    "MESSAGE": "high-availability: Can't update or insert instance. Retrying",
    "ICINGADB_ERROR": "can't perform \"INSERT INTO \\\"icingadb_instance\\\"...: Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction"
}

However, this retry mechanism also prevents new heartbeat events from being consumed from the channel and results in such weird situation.

{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 11:57:41",
    "ICINGADB_PREVIOUS": "2024-07-25 11:57:38.433795293 +0200 CEST",
    "ICINGADB_CURRENT": "2024-07-25 11:57:41.433328734 +0200 CEST",
    "MESSAGE": "heartbeat: Previous heartbeat not read from channel",
}

In order to prevent it from being retried endlessly, though, we have a deadline[^2] for the entire HA realize thing. https://github.com/Icinga/icingadb/blob/f9ce7a57bcb6a5b6b2ebfafafd8f91367e56bbf5/pkg/icingadb/ha.go#L220-L221

Unfortunately, if all this retry error mess then lasts for exactly one minute, and barely manages to survive the context deadline, it becomes in the end a wasted time, as after this delay the heartbeat of a given instance is already outdated, but has nevertheless been written to the database. Obviously, the current instance remains responsible, it just has written an outdated heartbeat timestamp to the database because of the deadlocks. When the passive instance finally reads that timestamp, it thinks that the other instance is no longer available and takes over the HA responsibility. https://github.com/Icinga/icingadb/blob/f9ce7a57bcb6a5b6b2ebfafafd8f91367e56bbf5/pkg/icingadb/ha.go#L319-L321

Having two active instances, however, does not produce any good results and brings down one of the two services.

Active IcingaDB Instance 
{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 11:58:32",
    "ICINGADB_PREVIOUS": "2024-07-25 11:58:29.433879058 +0200 CEST",
    "ICINGADB_CURRENT": "2024-07-25 11:58:32.433682676 +0200 CEST",
    "MESSAGE": "heartbeat: Previous heartbeat not read from channel",
},
{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 11:58:35",
    "ICINGADB_PREVIOUS": "2024-07-25 11:58:32.433682676 +0200 CEST",
    "ICINGADB_CURRENT": "2024-07-25 11:58:35.433895608 +0200 CEST",
    "MESSAGE": "heartbeat: Previous heartbeat not read from channel",
},
{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 11:58:38",
    "ICINGADB_PREVIOUS": "2024-07-25 11:58:35.433895608 +0200 CEST",
    "ICINGADB_CURRENT": "2024-07-25 11:58:38.432917241 +0200 CEST",
    "MESSAGE": "heartbeat: Previous heartbeat not read from channel",
},
{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 11:58:41",
    "MESSAGE": "high-availability: Instance updated/inserted successfully after error",
    "ICINGADB_AFTER": "1m9.144891965s",
    "ICINGADB_RECOVERED_ERROR": "can't perform \"INSERT INTO \\\"icingadb_instance\\\"...: Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction",
}

Take a closer look at the ICINGADB_AFTER field of the above log entry "ICINGADB_AFTER": "1m9.144891965s" originated from this part of the code. https://github.com/Icinga/icingadb/blob/f9ce7a57bcb6a5b6b2ebfafafd8f91367e56bbf5/pkg/icingadb/ha.go#L410-L413

The other(passive) Instance uses the heartbeat timestamp of this Instance from the database to determine that this currently active Instance is absent, by verifying that our timestamp does not exceed the 1m + grace period of 5s limit. However, this instance has spent over a minute and 5 seconds (to be precise, it took 1m9.144891965s) to record the heartbeat, so that it immediately exceeded the aforementioned limit right after the upsert. https://github.com/Icinga/icingadb/blob/f9ce7a57bcb6a5b6b2ebfafafd8f91367e56bbf5/pkg/icingadb/ha.go#L319 https://github.com/Icinga/icingadb/blob/f9ce7a57bcb6a5b6b2ebfafafd8f91367e56bbf5/pkg/icingadb/ha.go#L25-L28

Subsequently, the other Instance takes over the HA responsibility, even though this one is actually still responsible.

Passive IcingaDB Instance 
{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 11:58:33",
    "MESSAGE": "high-availability: Can't update or insert instance. Retrying",
    "ICINGADB_ERROR": "can't commit transaction: Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction"
},
{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 11:58:41",
    "ICINGADB_PREVIOUS": "2024-07-25 11:58:38.302404535 +0200 CEST",
    "ICINGADB_CURRENT": "2024-07-25 11:58:41.299007643 +0200 CEST",
    "MESSAGE": "heartbeat: Previous heartbeat not read from channel",
},
{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 11:58:43",
    "ICINGADB_HEARTBEAT": "2024-07-25 11:57:32.259000062 +0200 CEST",
    "ICINGADB_HEARTBEAT_AGE": "1m11.394978989s",
    "MESSAGE": "high-availability: Preparing to take over HA as other instance's heartbeat has expired",
},
{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 11:58:44",
    "ICINGADB_PREVIOUS": "2024-07-25 11:58:41.299007643 +0200 CEST",
    "ICINGADB_CURRENT": "2024-07-25 11:58:44.298509894 +0200 CEST",
    "MESSAGE": "heartbeat: Previous heartbeat not read from channel",
},
{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 11:59:15",
    "MESSAGE": "high-availability: Instance updated/inserted successfully after error",
    "ICINGADB_AFTER": "1m3.721497393s",
    "ICINGADB_RECOVERED_ERROR": "can't commit transaction: Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction",
},

Afterwards, the passive instance also becomes active and starts its initial config sync and encounters numerous duplicate key and deadlock errors until one of the queries exceeds the retry limit of 5m and crashes the Icinga DB process.

{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 11:59:20",
    "MESSAGE": "dump-signals: Received dump signal from Redis",
},
{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 12:00:20",
    "MESSAGE": "database: Can't execute query. Retrying",
    "ICINGADB_ERROR": "can't perform \"INSERT INTO \\\"host_customvar\\\"...: Error 1062 (23000): Duplicate entry '\\xDE\\xF7i\\xF9\\x9A\\xFEz\\xF6\\x8B\\x107\\xAAE\\xF0\\xBE\\xFF\\xA1Z\\xD3...' for key 'PRIMARY'"
},
{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 12:00:20",
    "MESSAGE": "database: Can't execute query. Retrying",
    "ICINGADB_ERROR": "can't perform \"INSERT INTO \\\"notification_customvar\\\"...: Error 1062 (23000): Duplicate entry '~\\x1E\\xEB\\x8B\\x8D\\xC2g\\x07\\xE4\\x8A\\xA0^\\x13\\x14\\xE0l\\xD0\\x0F\\...' for key 'PRIMARY'"
},
{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 12:00:20",
    "MESSAGE": "database: Can't execute query. Retrying",
    "ICINGADB_ERROR": "can't perform \"INSERT INTO \\\"state_history\\\"...: Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction"
},
{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 12:00:20",
    "MESSAGE": "database: Query retried successfully after error",
    "ICINGADB_AFTER": "833.661486ms",
    "ICINGADB_RECOVERED_ERROR": "can't perform \"INSERT INTO \\\"state_history\\\"...: Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction",
},
{
    "_SOURCE_REALTIME_TIMESTAMP": "2024-07-25 12:04:22",
    "MESSAGE": "Error 1062 (23000): Duplicate entry '\\xE9\\xF6\\xC3\\x12\\xA1\\xBE\\xBCo8`h\\xBCA3\\xD72rED\\xE0' for key 'PRIMARY'\ncan't perform \"INSERT INTO \\\"host_customvar\\\" (\\\"host_id\\\", \\\"customvar_id\\\", \\\"environment_id\\\", \\\"id\\\") VALUES (:host_id, :customvar_id, :environment_id, :id)\"\ngithub.com/icinga/icingadb/internal.CantPerformQuery\n\tgithub.com/icinga/icingadb/internal/internal.go:30\ngithub.com/icinga/icingadb/pkg/icingadb.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2.1\n\tgithub.com/icinga/icingadb/pkg/icingadb/db.go:412\ngithub.com/icinga/icingadb/pkg/retry.WithBackoff\n\tgithub.com/icinga/icingadb/pkg/retry/retry.go:60\ngithub.com/icinga/icingadb/pkg/icingadb.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2\n\tgithub.com/icinga/icingadb/pkg/icingadb/db.go:407\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1695\nretry deadline exceeded\ngithub.com/icinga/icingadb/pkg/retry.WithBackoff\n\tgithub.com/icinga/icingadb/pkg/retry/retry.go:95\ngithub.com/icinga/icingadb/pkg/icingadb.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2\n\tgithub.com/icinga/icingadb/pkg/icingadb/db.go:407\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1695",
}

ref/IP/48748

Additional context

Error 1062 (23000): Duplicate entry '\\xE9\\xF6\\xC3\\x12\\xA1\\xBE\\xBCo8`h\\xBCA3\\xD72rED\\xE0' for key 'PRIMARY'
can't perform \"INSERT INTO \"host_customvar\" (\"host_id\", \"customvar_id\", \"environment_id\", \"id\") VALUES (:host_id, :customvar_id, :environment_id, :id)\"
github.com/icinga/icingadb/internal.CantPerformQuery
    github.com/icinga/icingadb/internal/internal.go:30
github.com/icinga/icingadb/pkg/icingadb.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2.1
    github.com/icinga/icingadb/pkg/icingadb/db.go:412
github.com/icinga/icingadb/pkg/retry.WithBackoff
    github.com/icinga/icingadb/pkg/retry/retry.go:60
github.com/icinga/icingadb/pkg/icingadb.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2
    github.com/icinga/icingadb/pkg/icingadb/db.go:407
golang.org/x/sync/errgroup.(*Group).Go.func1
    golang.org/x/[email protected]/errgroup/errgroup.go:78
runtime.goexit
    runtime/asm_amd64.s:1695
retry deadline exceeded
github.com/icinga/icingadb/pkg/retry.WithBackoff
    github.com/icinga/icingadb/pkg/retry/retry.go:95
github.com/icinga/icingadb/pkg/icingadb.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2
    github.com/icinga/icingadb/pkg/icingadb/db.go:407
golang.org/x/sync/errgroup.(*Group).Go.func1
    golang.org/x/[email protected]/errgroup/errgroup.go:78
runtime.goexit
    runtime/asm_amd64.s:1695

[^1]: It's a well-known fact that the Icinga 2 heartbeats arrives in Redis every 1 second, but we only read them every 3 seconds and these 3 seconds are the pacesetter for HA realize thing. [^2]: The overall Redis heartbeat timeout is set to 1m, i.e. if we do not receive an Icinga 2 heartbeat for more than one minute, we hand over the HA responsibility. Seemingly, that very same timeout is also used as a deadline for the HA transaction retry's.

yhabteab avatar Aug 07 '24 16:08 yhabteab

In the last days, I was digging both into the HA realization logic and the context lifecycle.

Regarding HA.realize, I gave retry.WithBackoff another setting to strictly honor its context in Icinga/icinga-go-library#69. This changes was utilized in #800, allowing to directly bail out to the handover code.

In the context domain, there is unfortunately no direct relationship between the HA context and the used context for database interactions. Due to the architecture, it is not so easy to let all database queries honor a derived HA context, as I initially hoped when building Icinga/icinga-go-library#68 therefore.

Details about the derived contexts 

main.go:95   ctx w/ cancel from context.Background

main.go:109 heartbeat = icingaredis.NewHeartbeat(ctx, rc, logs.GetChildLogger("heartbeat"))
===
heartbeat.go:37 func NewHeartbeat(ctx context.Context, client *redis.Client, logger *logging.Logger) *Heartbeat {
heartbeat.go:48 go heartbeat.controller(ctx)
heartbeat.go:88 func (h *Heartbeat) controller(ctx context.Context) {
heartbeat.go:94 g, ctx := errgroup.WithContext(ctx)

From here on, the derived main context is used to abort the goroutine workers
and is given to the Redis read command for icinga:stats
===

main.go:116 ha = icingadb.NewHA(ctx, db, heartbeat, logs.GetChildLogger("high-availability"))
===
ha.go:57    func NewHA(ctx context.Context, db *database.DB, heartbeat *icingaredis.Heartbeat, logger *logging.Logger) *HA {
ha.go:58    ctx, cancelCtx := context.WithCancel(ctx)
ha.go:62    -> assigned to HA.ctx struct field
ha.go:220   realizeCtx, cancelRealizeCtx := context.WithDeadline(h.ctx, m.ExpiryTime())
ha.go:221   err = h.realize(realizeCtx, s, t, envId, shouldLogRoutineEvents)
ha.go:221   cancelRealizeCtx()
======
ha.go:280	if _, ok := ctx.Deadline(); !ok { panic("can't use context w/o deadline in realize()")
ha.go:284   err := retry.WithBackoff( ctx,

This context is being deadline-bound to m.ExpiryTime(), which is set to the
machine's current timestamp after being read from Redis plus an additional
minute. Thus, it should not exceed this minute.. should.
=========
ha:go:299   tx, errBegin := h.db.BeginTxx(ctx, &sql.TxOptions{Isolation: isoLvl})
ha.go:308   errQuery := tx.QueryRowxContext(ctx, query, envId, "y", h.instanceId).StructScan(instance)
ha.go:367   if _, err := tx.NamedExecContext(ctx, stmt, i); err != nil {
ha.go:373   _, err := tx.ExecContext(ctx, stmt, "n", envId, h.instanceId)

The derived context is used to check if another Icinga DB is responsible and
potentially proceeds to take over. All this is performed embedded in the retry
logic, being context-bound to the HA context.
=========
======
ha.go:442   if _, err := h.db.ExecContext(h.ctx, stmt, "n", h.instanceId); err != nil && !utils.IsContextCanceled(err) {
ha.go:453   if _, err := h.db.NamedExecContext(h.ctx, stmt, h.environment); err != nil {
ha.go:478   result, err := h.db.ExecContext(h.ctx, query, h.instanceId, envId, s.EndpointId, heartbeat)

The derived ha.ctx can be canceled either via ha.Close() or ha.abort().
- The abort() method may be called at multiple error handling cases within the
  ha.controller() goroutine.
- The Close() method is only called from a defer of the program's main function.

An internal abort() call stores an error next to closing the channel, being
retrieved in main.go:330, resulting to a logger.Fatalf().

In addition, there are some context checks to break out of selects.
===

main.go:119	telemetry.StartHeartbeat(ctx, rc, telemetryLogger, ha, heartbeat)
===
heartbeat.go:90 func StartHeartbeat(ctx context.Context, client *redis.Client, logger *logging.Logger, ha ha, heartbeat *icingaredis.Heartbeat,
heartbeat.go:100    periodic.Start(ctx, interval, func(tick periodic.Tick) {
heartbeat.go:127    cmd := client.XAdd(ctx, &redis.XAddArgs{

The context is used to terminate a periodic Redis insert job.
===

main.go:120	telemetry.WriteStats(ctx, rc, telemetryLogger)
===
stats.go:30 periodic.Start(ctx, time.Second, func(_ periodic.Tick) {
stats.go:39 cmd := client.XAdd(ctx, &redis.XAddArgs{

The context is used to terminate a periodic Redis insert job, again.
===

main.go:151 if err := hs.Sync(ctx); err != nil && !utils.IsContextCanceled(err) {
===
sync.go:40  func (s Sync) Sync(ctx context.Context) error {
sync.go:41  g, ctx := errgroup.WithContext(ctx)
sync.go:81  return s.readFromRedis(ctx, key, ch[0])
sync.go:89  return stage(ctx, s, key, ch[i], ch[i+1])
sync.go:94  return s.deleteFromRedis(ctx, key, ch[len(pipeline)])

This is the history sync in all its glory. The context will be passed to the
stage functions, which I am going to ignore mostly at this point. However,
there is at least
sync.go:261 return s.db.UpsertStreamed(ctx, insert, database.OnSuccessSendTo[database.Entity](inserted))
forwarding history events to their history database table.
===

main.go:158 hactx, cancelHactx := context.WithCancel(ctx)
===
main.go:159 for hactx.Err() == nil {
main.go:165 for hactx.Err() == nil {
main.go:166 synctx, cancelSynctx := context.WithCancel(ha.Environment().NewContext(hactx))
======
This synctx is now being used at 20 subsequent places. In particular at
main.go:227 return s.SyncAfterDump(synctx, common.NewSyncSubject(factory), dump)
where also NewHostCustomvar occurs, being the origin of the error posted at
https://github.com/Icinga/icingadb/issues/787

However, when handing over,
main.go:323 case handoverReason := <-ha.Handover():
the hactx is canceled, which is the parent of the synctx, effectively aborting
the just referenced faulty synchronization.
======
===

main.go:332 } else if ctx.Err() == nil {
main.go:340 case <-ctx.Done():

oxzi avatar Sep 05 '24 10:09 oxzi

I'm probably having the same issue with 2 icingadb instances and PostgreSQL backend. @oxzi Do you need more verbose logs?

1st instance

Nov 19 07:41:25 icingadb[2362535]: retention: Removed 5450 old state history items
Nov 19 07:41:31 icingadb[2362535]: overdue-sync: Synced 2 service overdue indicators
Nov 19 07:41:42 icingadb[2362535]: history-sync: Synced 34 state history items
Nov 19 07:41:44 icingadb[2362535]: runtime-updates: Upserted 34 ServiceState items
Nov 19 07:41:50 icingadb[2362535]: retention: Removed 141 old notification history items
Nov 19 07:41:51 icingadb[2362535]: overdue-sync: Synced 1 service overdue indicators
Nov 19 07:42:02 icingadb[2362535]: history-sync: Synced 27 state history items
Nov 19 07:42:04 icingadb[2362535]: runtime-updates: Upserted 35 ServiceState items
Nov 19 07:42:11 icingadb[2362535]: overdue-sync: Synced 1 service overdue indicators
Nov 19 07:42:22 icingadb[2362535]: history-sync: Synced 34 state history items
Nov 19 07:42:24 icingadb[2362535]: runtime-updates: Upserted 28 ServiceState items
Nov 19 07:42:42 icingadb[2362535]: history-sync: Synced 34 state history items
Nov 19 07:42:44 icingadb[2362535]: runtime-updates: Upserted 34 ServiceState items
Nov 19 07:43:02 icingadb[2362535]: history-sync: Synced 42 state history items
Nov 19 07:43:04 icingadb[2362535]: runtime-updates: Upserted 47 ServiceState items
Nov 19 07:43:11 icingadb[2362535]: overdue-sync: Synced 1 service overdue indicators
Nov 19 07:43:22 icingadb[2362535]: history-sync: Synced 44 state history items
Nov 19 07:43:24 icingadb[2362535]: runtime-updates: Upserted 43 ServiceState items
Nov 19 07:43:42 icingadb[2362535]: history-sync: Synced 2 notification history items
Nov 19 07:43:42 icingadb[2362535]: history-sync: Synced 42 state history items
Nov 19 07:43:44 icingadb[2362535]: runtime-updates: Upserted 43 ServiceState items
Nov 19 07:43:51 icingadb[2362535]: overdue-sync: Synced 1 service overdue indicators
Nov 19 07:44:02 icingadb[2362535]: history-sync: Synced 36 state history items
Nov 19 07:44:04 icingadb[2362535]: runtime-updates: Upserted 29 ServiceState items
Nov 19 07:44:22 icingadb[2362535]: history-sync: Synced 1 notification history items
Nov 19 07:44:22 icingadb[2362535]: history-sync: Synced 52 state history items
Nov 19 07:44:24 icingadb[2362535]: runtime-updates: Upserted 55 ServiceState items
Nov 19 07:44:31 icingadb[2362535]: overdue-sync: Synced 1 service overdue indicators
Nov 19 07:44:42 icingadb[2362535]: history-sync: Synced 42 state history items
Nov 19 07:44:44 icingadb[2362535]: runtime-updates: Upserted 47 ServiceState items
Nov 19 07:44:51 icingadb[2362535]: overdue-sync: Synced 1 service overdue indicators
Nov 19 07:45:02 icingadb[2362535]: history-sync: Synced 70 state history items
Nov 19 07:45:04 icingadb[2362535]: runtime-updates: Upserted 66 ServiceState items
Nov 19 07:45:22 icingadb[2362535]: history-sync: Synced 41 state history items
Nov 19 07:45:24 icingadb[2362535]: runtime-updates: Upserted 38 ServiceState items
Nov 19 07:45:42 icingadb[2362535]: history-sync: Synced 39 state history items
Nov 19 07:45:44 icingadb[2362535]: runtime-updates: Upserted 41 ServiceState items
Nov 19 07:46:02 icingadb[2362535]: history-sync: Synced 37 state history items
Nov 19 07:46:04 icingadb[2362535]: runtime-updates: Upserted 35 ServiceState items
Nov 19 07:46:11 icingadb[2362535]: overdue-sync: Synced 1 service overdue indicators
Nov 19 07:46:22 icingadb[2362535]: history-sync: Synced 35 state history items
Nov 19 07:46:22 icingadb[2362535]: history-sync: Synced 1 notification history items
Nov 19 07:46:24 icingadb[2362535]: runtime-updates: Upserted 42 ServiceState items
Nov 19 07:46:31 icingadb[2362535]: overdue-sync: Synced 1 service overdue indicators
Nov 19 07:46:42 icingadb[2362535]: history-sync: Synced 46 state history items

2nd instance

Nov 19 07:43:11 icingadb[3899473]: history-sync: Synced 41 state history items
Nov 19 07:43:31 icingadb[3899473]: history-sync: Synced 53 state history items
Nov 19 07:43:31 icingadb[3899473]: high-availability: Another instance is active
Nov 19 07:43:51 icingadb[3899473]: history-sync: Synced 2 notification history items
Nov 19 07:43:51 icingadb[3899473]: history-sync: Synced 33 state history items
Nov 19 07:44:11 icingadb[3899473]: history-sync: Synced 33 state history items
Nov 19 07:44:31 icingadb[3899473]: history-sync: Synced 1 notification history items
Nov 19 07:44:31 icingadb[3899473]: history-sync: Synced 57 state history items
Nov 19 07:44:51 icingadb[3899473]: history-sync: Synced 58 state history items
Nov 19 07:45:11 icingadb[3899473]: history-sync: Synced 52 state history items
Nov 19 07:45:31 icingadb[3899473]: history-sync: Synced 40 state history items
Nov 19 07:45:51 icingadb[3899473]: history-sync: Synced 39 state history items
Nov 19 07:46:11 icingadb[3899473]: history-sync: Synced 1 notification history items
Nov 19 07:46:11 icingadb[3899473]: history-sync: Synced 34 state history items
Nov 19 07:46:31 icingadb[3899473]: history-sync: Synced 41 state history items
Nov 19 07:46:51 icingadb[3899473]: history-sync: Synced 1 notification history items
Nov 19 07:46:51 icingadb[3899473]: history-sync: Synced 40 state history items
Nov 19 07:47:11 icingadb[3899473]: history-sync: Synced 28 state history items
Nov 19 07:47:31 icingadb[3899473]: history-sync: Synced 1 notification history items
Nov 19 07:47:31 icingadb[3899473]: history-sync: Synced 34 state history items
Nov 19 07:47:51 icingadb[3899473]: history-sync: Synced 41 state history items
Nov 19 07:48:11 icingadb[3899473]: history-sync: Synced 41 state history items
Nov 19 07:48:31 icingadb[3899473]: high-availability: Another instance is active
Nov 19 07:48:31 icingadb[3899473]: history-sync: Synced 81 state history items
Nov 19 07:48:51 icingadb[3899473]: history-sync: Synced 65 state history items
Nov 19 07:49:11 icingadb[3899473]: history-sync: Synced 56 state history items
Nov 19 07:49:31 icingadb[3899473]: history-sync: Synced 46 state history items
Nov 19 07:49:51 icingadb[3899473]: history-sync: Synced 37 state history items
Nov 19 07:50:11 icingadb[3899473]: history-sync: Synced 32 state history items
Nov 19 07:50:31 icingadb[3899473]: history-sync: Synced 1 notification history items
Nov 19 07:50:31 icingadb[3899473]: history-sync: Synced 38 state history items
Nov 19 07:50:51 icingadb[3899473]: history-sync: Synced 33 state history items
Nov 19 07:51:11 icingadb[3899473]: history-sync: Synced 40 state history items
Nov 19 07:51:31 icingadb[3899473]: history-sync: Synced 43 state history items
Nov 19 07:51:51 icingadb[3899473]: history-sync: Synced 28 state history items
Nov 19 07:52:11 icingadb[3899473]: history-sync: Synced 1 notification history items
Nov 19 07:52:11 icingadb[3899473]: history-sync: Synced 28 state history items
Nov 19 07:52:31 icingadb[3899473]: history-sync: Synced 41 state history items
Nov 19 07:52:51 icingadb[3899473]: history-sync: Synced 49 state history items
Nov 19 07:53:11 icingadb[3899473]: history-sync: Synced 41 state history items
Nov 19 07:53:31 icingadb[3899473]: high-availability: Another instance is active
Nov 19 07:53:31 icingadb[3899473]: history-sync: Synced 38 state history items
Nov 19 07:53:51 icingadb[3899473]: history-sync: Synced 26 state history items
Nov 19 07:54:11 icingadb[3899473]: history-sync: Synced 29 state history items
Nov 19 07:54:31 icingadb[3899473]: history-sync: Synced 45 state history items
Nov 19 07:54:51 icingadb[3899473]: history-sync: Synced 1 notification history items
Nov 19 07:54:51 icingadb[3899473]: history-sync: Synced 49 state history items
Nov 19 07:55:11 icingadb[3899473]: history-sync: Synced 45 state history items
Nov 19 07:55:31 icingadb[3899473]: history-sync: Synced 56 state history items
Nov 19 07:55:32 icingadb[3899473]: database: Can't execute query. Retrying
Nov 19 07:55:32 icingadb[3899473]: database: Query retried successfully after error
Nov 19 07:55:51 icingadb[3899473]: history-sync: Synced 48 state history items
Nov 19 07:56:11 icingadb[3899473]: history-sync: Synced 46 state history items
Nov 19 07:56:31 icingadb[3899473]: history-sync: Synced 32 state history items
Nov 19 07:56:51 icingadb[3899473]: history-sync: Synced 52 state history items
Nov 19 07:57:11 icingadb[3899473]: history-sync: Synced 82 state history items
Nov 19 07:57:17 icingadb[3899473]: database: Can't execute query. Retrying
Nov 19 07:57:20 icingadb[3899473]: database: Query retried successfully after error
Nov 19 07:57:21 icingadb[3899473]: database: Can't execute query. Retrying
Nov 19 07:57:21 icingadb[3899473]: database: Can't execute query. Retrying
Nov 19 07:57:21 icingadb[3899473]: database: Can't execute query. Retrying
Nov 19 07:57:24 icingadb[3899473]: high-availability: Can't update or insert instance. Retrying
Nov 19 07:57:26 icingadb[3899473]: high-availability: Can't update or insert instance. Retrying
Nov 19 07:57:29 icingadb[3899473]: high-availability: Can't update or insert instance. Retrying
Nov 19 07:57:31 icingadb[3899473]: history-sync: Synced 19 state history items
Nov 19 07:57:32 icingadb[3899473]: high-availability: Can't update or insert instance. Retrying
Nov 19 07:57:34 icingadb[3899473]: high-availability: Can't update or insert instance. Retrying
Nov 19 07:57:36 icingadb[3899473]: high-availability: Can't update or insert instance. Retrying
Nov 19 07:57:38 icingadb[3899473]: high-availability: Can't update or insert instance. Retrying
Nov 19 07:57:40 icingadb[3899473]: high-availability: Can't update or insert instance. Retrying
Nov 19 07:57:42 icingadb[3899473]: high-availability: Can't update or insert instance. Retrying
Nov 19 07:57:45 icingadb[3899473]: high-availability: Can't update or insert instance. Retrying
Nov 19 07:57:47 icingadb[3899473]: high-availability: Can't update or insert instance. Retrying
Nov 19 07:57:48 icingadb[3899473]: high-availability: Can't update or insert instance. Retrying
Nov 19 07:57:50 icingadb[3899473]: high-availability: Can't update or insert instance. Retrying
Nov 19 08:01:12 icingadb[3899473]: pq: could not serialize access due to read/write dependencies among transactions
                                   can't perform "INSERT INTO \"icingadb_instance\" (\"id\", \"icinga2_active_service_checks_enabled\", \"icinga2_notifications_enabled\", \"icinga2_flap_detection_enabled\", \"icinga2_event_handlers_enabled\", \"icinga2_version\", \"endpoint_id\", \"heartbeat\", \"environment_id\", \"icinga2_active_host_checks_enabled\", \"icinga2_performance_data_enabled\", \"responsible\", \"icinga2_start_time\") VALUES (:id,:icinga2_active_service_checks_enabled,:icinga2_notifications_enabled,:icinga2_flap_detection_enabled,:icinga2_event_handlers_enabled,:icinga2_version,:endpoint_id,:heartbeat,:environment_id,:icinga2_active_host_checks_enabled,:icinga2_performance_data_enabled,:responsible,:icinga2_start_time) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \"id\" = EXCLUDED.\"id\",\"icinga2_active_service_checks_enabled\" = EXCLUDED.\"icinga2_active_service_checks_enabled\",\"icinga2_notifications_enabled\" = EXCLUDED.\"icinga2_notifications_enabled\",\"icinga2_flap_detection_enabled\" = EXCLUDED.\"icinga2_flap_detection_enabled\",\"icinga2_event_handlers_enabled\" = EXCLUDED.\"icinga2_event_handlers_enabled\",\"icinga2_version\" = EXCLUDED.\"icinga2_version\",\"endpoint_id\" = EXCLUDED.\"endpoint_id\",\"heartbeat\" = EXCLUDED.\"heartbeat\",\"environment_id\" = EXCLUDED.\"environment_id\",\"icinga2_active_host_checks_enabled\" = EXCLUDED.\"icinga2_active_host_checks_enabled\",\"icinga2_performance_data_enabled\" = EXCLUDED.\"icinga2_performance_data_enabled\",\"responsible\" = EXCLUDED.\"responsible\",\"icinga2_start_time\" = EXCLUDED.\"icinga2_start_time\""
                                   github.com/icinga/icingadb/internal.CantPerformQuery
                                           github.com/icinga/icingadb/internal/internal.go:30
                                   github.com/icinga/icingadb/pkg/icingadb.(*HA).realize.func1
                                           github.com/icinga/icingadb/pkg/icingadb/ha.go:368
                                   github.com/icinga/icingadb/pkg/retry.WithBackoff
                                           github.com/icinga/icingadb/pkg/retry/retry.go:60
                                   github.com/icinga/icingadb/pkg/icingadb.(*HA).realize
                                           github.com/icinga/icingadb/pkg/icingadb/ha.go:284
                                   github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                           github.com/icinga/icingadb/pkg/icingadb/ha.go:221
                                   runtime.goexit
                                           runtime/asm_amd64.s:1695
                                   HA aborted
                                   github.com/icinga/icingadb/pkg/icingadb.(*HA).abort.func1
                                           github.com/icinga/icingadb/pkg/icingadb/ha.go:131
                                   sync.(*Once).doSlow
                                           sync/once.go:74
                                   sync.(*Once).Do
                                           sync/once.go:65
                                   github.com/icinga/icingadb/pkg/icingadb.(*HA).abort
                                           github.com/icinga/icingadb/pkg/icingadb/ha.go:129
                                   github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                           github.com/icinga/icingadb/pkg/icingadb/ha.go:233
                                   runtime.goexit
                                           runtime/asm_amd64.s:1695
                                   HA exited with an error
                                   main.run
                                           github.com/icinga/icingadb/cmd/icingadb/main.go:335
                                   main.main
                                           github.com/icinga/icingadb/cmd/icingadb/main.go:37
                                   runtime.main
                                           runtime/proc.go:271
                                   runtime.goexit
                                           runtime/asm_amd64.s:1695
Nov 19 08:01:12 systemd[1]: icingadb.service: Main process exited, code=exited, status=1/FAILURE
Nov 19 08:01:12 systemd[1]: icingadb.service: Failed with result 'exit-code'.

deric avatar Nov 19 '24 08:11 deric

Hi @deric, thanks for reporting!

Your logs actually look fine except for the last crash. Note that both instances are allowed to write history data concurrently, i.e. while the active instance logs Nov 19 07:46:24 icingadb[2362535]: runtime-updates: Upserted 42 ServiceState items in addition to history and overdue states, the passive one only logs history-related stuff and that is as expected. The last crash is not nice though, but has a different reason which will be fixed by #830 (see the PostgreSQL Tests cases) and https://github.com/Icinga/icinga-go-library/pull/59.

yhabteab avatar Nov 19 '24 09:11 yhabteab

@deric: Thanks for posting your logs and please excuse the trouble.

If possible, please supply the logs with additional fields via journalctl's JSON output (-o json). Having debug logs would also be very helpful.

Even if #800 also has a MySQL spin, it addresses some general HA realization topics. Is it possible to deploy a custom build in your environment? There is also #830 with more PostgreSQL specifics.

How often does this happen? Was this the first time or it this a more occasional event? Was there any additional load to the database during the event?

oxzi avatar Nov 19 '24 09:11 oxzi

@oxzi It happens quite frequently (e.g. daily), the database runs on a dedicated physical server. There's basically no load at all until queries from both icingadb cause a deadlock.

icingadb version v1.2.0

Yes, I can deploy a custom build. Do you have pre-release packages or should I compile it from a specific branch?

yet another crash:

Nov 19 13:36:48 icingadb[4121746]: high-availability: Can't update or insert instance. Retrying
Nov 19 13:36:50 icingadb[4121746]: high-availability: Can't update or insert instance. Retrying
Nov 19 13:36:51 icingadb[4121746]: high-availability: Can't update or insert instance. Retrying
Nov 19 13:36:54 icingadb[4121746]: high-availability: Can't update or insert instance. Retrying
Nov 19 13:36:57 icingadb[4121746]: high-availability: Can't update or insert instance. Retrying
Nov 19 13:37:00 icingadb[4121746]: high-availability: Can't update or insert instance. Retrying
Nov 19 13:37:02 icingadb[4121746]: high-availability: Can't update or insert instance. Retrying
Nov 19 13:37:04 icingadb[4121746]: high-availability: Can't update or insert instance. Retrying
Nov 19 13:37:07 icingadb[4121746]: high-availability: Can't update or insert instance. Retrying
Nov 19 13:37:36 icingadb[4121746]: retry deadline exceeded
                                   github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                           github.com/icinga/icingadb/pkg/icingadb/ha.go:163
                                   runtime.goexit
                                           runtime/asm_amd64.s:1695
                                   HA aborted
                                   github.com/icinga/icingadb/pkg/icingadb.(*HA).abort.func1
                                           github.com/icinga/icingadb/pkg/icingadb/ha.go:131
                                   sync.(*Once).doSlow
                                           sync/once.go:74
                                   sync.(*Once).Do
                                           sync/once.go:65
                                   github.com/icinga/icingadb/pkg/icingadb.(*HA).abort
                                           github.com/icinga/icingadb/pkg/icingadb/ha.go:129
                                   github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                           github.com/icinga/icingadb/pkg/icingadb/ha.go:163
                                   runtime.goexit
                                           runtime/asm_amd64.s:1695
                                   HA exited with an error
                                   main.run
                                           github.com/icinga/icingadb/cmd/icingadb/main.go:335
                                   main.main
                                           github.com/icinga/icingadb/cmd/icingadb/main.go:37
                                   runtime.main
                                           runtime/proc.go:271
                                   runtime.goexit
                                           runtime/asm_amd64.s:1695
Nov 19 13:37:36 systemd[1]: icingadb.service: Main process exited, code=exited, status=1/FAILURE
Nov 19 13:37:36 systemd[1]: icingadb.service: Failed with result 'exit-code'.

-o json

{"_SYSTEMD_SLICE":"system.slice","_CAP_EFFECTIVE":"0","MESSAGE":"high-availability: Can't update or insert instance. Retrying","__MONOTONIC_TIMESTAMP":"2700083125870","_GID":"995","_SYSTEMD_UNIT":"icingadb.service","_HOSTNAME":"icm02","_TRANSPORT":"journal","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_PID":"4121746","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","__REALTIME_TIMESTAMP":"1732023356107751","_UID":"999","_SELINUX_CONTEXT":"unconfined\n","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","_COMM":"icingadb","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d0278a;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274a97ede6e;t=627441ce22be7;x=6c475791185bfc03","_RUNTIME_SCOPE":"system","_SOURCE_REALTIME_TIMESTAMP":"1732023356107726","_EXE":"/usr/sbin/icingadb","PRIORITY":"6","SYSLOG_IDENTIFIER":"icingadb","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"icinga2_performance_data_enabled\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"environment_id\\\", \\\"heartbeat\\\", \\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\", \\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\", \\\"id\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\") VALUES (:icinga2_performance_data_enabled,:responsible,:icinga2_start_time,:icinga2_flap_detection_enabled,:environment_id,:heartbeat,:endpoint_id,:icinga2_active_host_checks_enabled,:icinga2_notifications_enabled,:icinga2_version,:id,:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\",\\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\"\": pq: could not serialize access due to read/write dependencies among transactions"}
{"_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","PRIORITY":"6","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","__MONOTONIC_TIMESTAMP":"2700086117599","_EXE":"/usr/sbin/icingadb","SYSLOG_IDENTIFIER":"icingadb","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_UID":"999","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d0278c;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274a9ac84df;t=627441d0fd258;x=195aae6f1d217d9a","_SOURCE_REALTIME_TIMESTAMP":"1732023359099456","__REALTIME_TIMESTAMP":"1732023359099480","_PID":"4121746","_SYSTEMD_SLICE":"system.slice","MESSAGE":"high-availability: Can't update or insert instance. Retrying","_SELINUX_CONTEXT":"unconfined\n","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"icinga2_version\\\", \\\"id\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"heartbeat\\\", \\\"environment_id\\\", \\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\", \\\"icinga2_notifications_enabled\\\") VALUES (:icinga2_version,:id,:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled,:responsible,:icinga2_start_time,:icinga2_performance_data_enabled,:icinga2_flap_detection_enabled,:heartbeat,:environment_id,:endpoint_id,:icinga2_active_host_checks_enabled,:icinga2_notifications_enabled) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\",\\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\",\\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\"\": pq: could not serialize access due to read/write dependencies among transactions","_GID":"995","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","_COMM":"icingadb","_SYSTEMD_UNIT":"icingadb.service","_CAP_EFFECTIVE":"0","_TRANSPORT":"journal","_RUNTIME_SCOPE":"system","_HOSTNAME":"icm02"}
{"_SYSTEMD_SLICE":"system.slice","_COMM":"icingadb","_CAP_EFFECTIVE":"0","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","__REALTIME_TIMESTAMP":"1732023361075913","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","_SOURCE_REALTIME_TIMESTAMP":"1732023361075804","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\", \\\"id\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"heartbeat\\\", \\\"environment_id\\\", \\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\") VALUES (:icinga2_notifications_enabled,:icinga2_version,:id,:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled,:responsible,:icinga2_start_time,:icinga2_performance_data_enabled,:icinga2_flap_detection_enabled,:heartbeat,:environment_id,:endpoint_id,:icinga2_active_host_checks_enabled) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\",\\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\"\": pq: could not serialize access due to read/write dependencies among transactions","_SYSTEMD_UNIT":"icingadb.service","_PID":"4121746","MESSAGE":"high-availability: Can't update or insert instance. Retrying","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","_SELINUX_CONTEXT":"unconfined\n","_UID":"999","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d0278d;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274a9caad50;t=627441d2dfac9;x=d7944736533b9f70","_HOSTNAME":"icm02","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","__MONOTONIC_TIMESTAMP":"2700088094032","_EXE":"/usr/sbin/icingadb","_TRANSPORT":"journal","SYSLOG_IDENTIFIER":"icingadb","_GID":"995","PRIORITY":"6","_RUNTIME_SCOPE":"system"}
{"_UID":"999","__MONOTONIC_TIMESTAMP":"2700090114123","_EXE":"/usr/sbin/icingadb","_HOSTNAME":"icm02","_TRANSPORT":"journal","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\", \\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"id\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"heartbeat\\\", \\\"environment_id\\\") VALUES (:endpoint_id,:icinga2_active_host_checks_enabled,:icinga2_notifications_enabled,:icinga2_version,:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled,:id,:responsible,:icinga2_start_time,:icinga2_performance_data_enabled,:icinga2_flap_detection_enabled,:heartbeat,:environment_id) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\",\\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\"\": pq: could not serialize access due to read/write dependencies among transactions","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","PRIORITY":"6","_RUNTIME_SCOPE":"system","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d0278e;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274a9e9804b;t=627441d4ccdc4;x=7c3cafd2fff4bfd1","_PID":"4121746","_GID":"995","_SELINUX_CONTEXT":"unconfined\n","_COMM":"icingadb","MESSAGE":"high-availability: Can't update or insert instance. Retrying","_SYSTEMD_UNIT":"icingadb.service","SYSLOG_IDENTIFIER":"icingadb","_SYSTEMD_SLICE":"system.slice","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","_CAP_EFFECTIVE":"0","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","_SOURCE_REALTIME_TIMESTAMP":"1732023363095994","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","__REALTIME_TIMESTAMP":"1732023363096004"}
{"_SELINUX_CONTEXT":"unconfined\n","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","_UID":"999","_HOSTNAME":"icm02","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"icinga2_version\\\", \\\"id\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"icinga2_performance_data_enabled\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"environment_id\\\", \\\"heartbeat\\\", \\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\", \\\"icinga2_notifications_enabled\\\") VALUES (:icinga2_version,:id,:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled,:icinga2_performance_data_enabled,:responsible,:icinga2_start_time,:icinga2_flap_detection_enabled,:environment_id,:heartbeat,:endpoint_id,:icinga2_active_host_checks_enabled,:icinga2_notifications_enabled) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\",\\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\"\": pq: could not serialize access due to read/write dependencies among transactions","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d0278f;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274aa08e7aa;t=627441d6c3523;x=1767b5c3df7f7ddd","_SYSTEMD_UNIT":"icingadb.service","__MONOTONIC_TIMESTAMP":"2700092172202","_TRANSPORT":"journal","MESSAGE":"high-availability: Can't update or insert instance. Retrying","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","_PID":"4121746","_EXE":"/usr/sbin/icingadb","_SYSTEMD_SLICE":"system.slice","_SOURCE_REALTIME_TIMESTAMP":"1732023365154066","SYSLOG_IDENTIFIER":"icingadb","PRIORITY":"6","_GID":"995","__REALTIME_TIMESTAMP":"1732023365154083","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_CAP_EFFECTIVE":"0","_RUNTIME_SCOPE":"system","_COMM":"icingadb"}
{"__MONOTONIC_TIMESTAMP":"2700126464512","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_TRANSPORT":"journal","_SELINUX_CONTEXT":"unconfined\n","MESSAGE":"high-availability: Can't update or insert instance. Retrying","_RUNTIME_SCOPE":"system","_SYSTEMD_SLICE":"system.slice","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","__REALTIME_TIMESTAMP":"1732023399446393","_GID":"995","PRIORITY":"6","_SOURCE_REALTIME_TIMESTAMP":"1732023399446323","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","_CAP_EFFECTIVE":"0","_PID":"4121746","_UID":"999","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d02791;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274ac142a00;t=627441f777779;x=d703a87fe53c2bdc","_HOSTNAME":"icm02","_EXE":"/usr/sbin/icingadb","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"id\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"heartbeat\\\", \\\"environment_id\\\", \\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\", \\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\") VALUES (:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled,:id,:responsible,:icinga2_start_time,:icinga2_performance_data_enabled,:icinga2_flap_detection_enabled,:heartbeat,:environment_id,:endpoint_id,:icinga2_active_host_checks_enabled,:icinga2_notifications_enabled,:icinga2_version) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\",\\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\",\\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\"\": pq: could not serialize access due to read/write dependencies among transactions","SYSLOG_IDENTIFIER":"icingadb","_COMM":"icingadb","_SYSTEMD_UNIT":"icingadb.service","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml"}
{"_PID":"4121746","_RUNTIME_SCOPE":"system","__REALTIME_TIMESTAMP":"1732023401658647","_COMM":"icingadb","_SOURCE_REALTIME_TIMESTAMP":"1732023401658617","_SELINUX_CONTEXT":"unconfined\n","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","_TRANSPORT":"journal","__MONOTONIC_TIMESTAMP":"2700128676767","_HOSTNAME":"icm02","_SYSTEMD_SLICE":"system.slice","_CAP_EFFECTIVE":"0","SYSLOG_IDENTIFIER":"icingadb","_UID":"999","PRIORITY":"6","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"icinga2_version\\\", \\\"id\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"heartbeat\\\", \\\"environment_id\\\", \\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\", \\\"icinga2_notifications_enabled\\\") VALUES (:icinga2_version,:id,:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled,:responsible,:icinga2_start_time,:icinga2_performance_data_enabled,:icinga2_flap_detection_enabled,:heartbeat,:environment_id,:endpoint_id,:icinga2_active_host_checks_enabled,:icinga2_notifications_enabled) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\",\\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\",\\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\"\": pq: could not serialize access due to read/write dependencies among transactions","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_SYSTEMD_UNIT":"icingadb.service","MESSAGE":"high-availability: Can't update or insert instance. Retrying","_GID":"995","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","_EXE":"/usr/sbin/icingadb","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d02792;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274ac35eb9f;t=627441f993917;x=8981f3d2e5edf1d9"}
{"_HOSTNAME":"icm02","_SOURCE_REALTIME_TIMESTAMP":"1732023403533050","_UID":"999","_PID":"4121746","_RUNTIME_SCOPE":"system","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"responsible\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"environment_id\\\", \\\"heartbeat\\\", \\\"icinga2_active_host_checks_enabled\\\", \\\"endpoint_id\\\", \\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"id\\\", \\\"icinga2_active_service_checks_enabled\\\") VALUES (:icinga2_start_time,:icinga2_performance_data_enabled,:responsible,:icinga2_flap_detection_enabled,:environment_id,:heartbeat,:icinga2_active_host_checks_enabled,:endpoint_id,:icinga2_notifications_enabled,:icinga2_version,:icinga2_event_handlers_enabled,:id,:icinga2_active_service_checks_enabled) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\",\\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\"\": pq: could not serialize access due to read/write dependencies among transactions","SYSLOG_IDENTIFIER":"icingadb","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","__REALTIME_TIMESTAMP":"1732023403533081","_SYSTEMD_SLICE":"system.slice","__MONOTONIC_TIMESTAMP":"2700130551200","_SELINUX_CONTEXT":"unconfined\n","_SYSTEMD_UNIT":"icingadb.service","_EXE":"/usr/sbin/icingadb","PRIORITY":"6","_CAP_EFFECTIVE":"0","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d02793;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274ac5285a0;t=627441fb5d319;x=f814b25db4e2d8e7","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_GID":"995","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","_COMM":"icingadb","MESSAGE":"high-availability: Can't update or insert instance. Retrying","_TRANSPORT":"journal"}
{"_SYSTEMD_CGROUP":"/system.slice/icingadb.service","_HOSTNAME":"icm02","_SOURCE_REALTIME_TIMESTAMP":"1732023406336802","__REALTIME_TIMESTAMP":"1732023406336816","_PID":"4121746","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","MESSAGE":"high-availability: Can't update or insert instance. Retrying","PRIORITY":"6","_CAP_EFFECTIVE":"0","_RUNTIME_SCOPE":"system","_TRANSPORT":"journal","_GID":"995","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","_EXE":"/usr/sbin/icingadb","_SYSTEMD_UNIT":"icingadb.service","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\", \\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"id\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"heartbeat\\\", \\\"environment_id\\\") VALUES (:endpoint_id,:icinga2_active_host_checks_enabled,:icinga2_notifications_enabled,:icinga2_version,:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled,:id,:responsible,:icinga2_start_time,:icinga2_performance_data_enabled,:icinga2_flap_detection_enabled,:heartbeat,:environment_id) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\",\\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\"\": pq: could not serialize access due to read/write dependencies among transactions","_SYSTEMD_SLICE":"system.slice","_COMM":"icingadb","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_UID":"999","_SELINUX_CONTEXT":"unconfined\n","__MONOTONIC_TIMESTAMP":"2700133354935","SYSLOG_IDENTIFIER":"icingadb","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d02794;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274ac7d4db7;t=627441fe09b30;x=bf4443ce30751bf4"}
{"_HOSTNAME":"icm02","SYSLOG_IDENTIFIER":"icingadb","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","PRIORITY":"6","_TRANSPORT":"journal","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\", \\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\", \\\"id\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"heartbeat\\\", \\\"environment_id\\\") VALUES (:endpoint_id,:icinga2_active_host_checks_enabled,:icinga2_notifications_enabled,:icinga2_version,:id,:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled,:responsible,:icinga2_start_time,:icinga2_performance_data_enabled,:icinga2_flap_detection_enabled,:heartbeat,:environment_id) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\",\\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\"\": pq: could not serialize access due to read/write dependencies among transactions","_UID":"999","_GID":"995","__MONOTONIC_TIMESTAMP":"2700135275660","_PID":"4121746","_SELINUX_CONTEXT":"unconfined\n","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","MESSAGE":"high-availability: Can't update or insert instance. Retrying","_SYSTEMD_UNIT":"icingadb.service","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d02795;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274ac9a9c8c;t=627441ffdea05;x=ca9813d0c7774f6f","_COMM":"icingadb","_EXE":"/usr/sbin/icingadb","_RUNTIME_SCOPE":"system","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","_CAP_EFFECTIVE":"0","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","_SYSTEMD_SLICE":"system.slice","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","__REALTIME_TIMESTAMP":"1732023408257541","_SOURCE_REALTIME_TIMESTAMP":"1732023408257510"}
{"_UID":"999","_RUNTIME_SCOPE":"system","_HOSTNAME":"icm02","_SYSTEMD_SLICE":"system.slice","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_SOURCE_REALTIME_TIMESTAMP":"1732023410080176","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","_COMM":"icingadb","MESSAGE":"high-availability: Can't update or insert instance. Retrying","SYSLOG_IDENTIFIER":"icingadb","_PID":"4121746","_TRANSPORT":"journal","_SELINUX_CONTEXT":"unconfined\n","_SYSTEMD_UNIT":"icingadb.service","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\", \\\"id\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"heartbeat\\\", \\\"environment_id\\\", \\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\") VALUES (:icinga2_notifications_enabled,:icinga2_version,:id,:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled,:responsible,:icinga2_start_time,:icinga2_performance_data_enabled,:icinga2_flap_detection_enabled,:heartbeat,:environment_id,:endpoint_id,:icinga2_active_host_checks_enabled) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\",\\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\"\": pq: could not serialize access due to read/write dependencies among transactions","_EXE":"/usr/sbin/icingadb","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","_CAP_EFFECTIVE":"0","PRIORITY":"6","_GID":"995","__MONOTONIC_TIMESTAMP":"2700137098321","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","__REALTIME_TIMESTAMP":"1732023410080202","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d02796;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274acb66c51;t=627442019b9ca;x=a91a04bcae96edc0"}
{"PRIORITY":"6","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d02799;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274accfa2ee;t=627442032f067;x=bbf23fa6351b81d8","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","_COMM":"icingadb","_UID":"999","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_CAP_EFFECTIVE":"0","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\", \\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"id\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"heartbeat\\\", \\\"environment_id\\\") VALUES (:endpoint_id,:icinga2_active_host_checks_enabled,:icinga2_notifications_enabled,:icinga2_version,:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled,:id,:responsible,:icinga2_start_time,:icinga2_performance_data_enabled,:icinga2_flap_detection_enabled,:heartbeat,:environment_id) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\",\\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\"\": pq: could not serialize access due to read/write dependencies among transactions","_HOSTNAME":"icm02","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","_RUNTIME_SCOPE":"system","_GID":"995","SYSLOG_IDENTIFIER":"icingadb","__REALTIME_TIMESTAMP":"1732023411732583","_PID":"4121746","MESSAGE":"high-availability: Can't update or insert instance. Retrying","_SYSTEMD_UNIT":"icingadb.service","_TRANSPORT":"journal","_SYSTEMD_SLICE":"system.slice","_SELINUX_CONTEXT":"unconfined\n","__MONOTONIC_TIMESTAMP":"2700138750702","_SOURCE_REALTIME_TIMESTAMP":"1732023411732573","_EXE":"/usr/sbin/icingadb"}
{"_PID":"4121746","_SYSTEMD_UNIT":"icingadb.service","PRIORITY":"6","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","_CAP_EFFECTIVE":"0","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d0279b;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274acfc6213;t=62744205faf8c;x=714aa687523118c5","_UID":"999","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_SELINUX_CONTEXT":"unconfined\n","_EXE":"/usr/sbin/icingadb","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","_GID":"995","_RUNTIME_SCOPE":"system","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\", \\\"id\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"heartbeat\\\", \\\"environment_id\\\", \\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\") VALUES (:icinga2_notifications_enabled,:icinga2_version,:id,:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled,:responsible,:icinga2_start_time,:icinga2_performance_data_enabled,:icinga2_flap_detection_enabled,:heartbeat,:environment_id,:endpoint_id,:icinga2_active_host_checks_enabled) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\",\\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\"\": pq: could not serialize access due to read/write dependencies among transactions","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","MESSAGE":"high-availability: Can't update or insert instance. Retrying","_HOSTNAME":"icm02","SYSLOG_IDENTIFIER":"icingadb","_SYSTEMD_SLICE":"system.slice","__MONOTONIC_TIMESTAMP":"2700141683219","_COMM":"icingadb","_TRANSPORT":"journal","__REALTIME_TIMESTAMP":"1732023414665100","_SOURCE_REALTIME_TIMESTAMP":"1732023414665083"}
{"_PID":"4121746","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","MESSAGE":"high-availability: Can't update or insert instance. Retrying","_UID":"999","_EXE":"/usr/sbin/icingadb","_SELINUX_CONTEXT":"unconfined\n","_COMM":"icingadb","_HOSTNAME":"icm02","_SOURCE_REALTIME_TIMESTAMP":"1732023417498551","_SYSTEMD_SLICE":"system.slice","_CAP_EFFECTIVE":"0","_GID":"995","PRIORITY":"6","__MONOTONIC_TIMESTAMP":"2700144516680","_RUNTIME_SCOPE":"system","SYSLOG_IDENTIFIER":"icingadb","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"icinga2_event_handlers_enabled\\\", \\\"id\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"responsible\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"environment_id\\\", \\\"heartbeat\\\", \\\"icinga2_active_host_checks_enabled\\\", \\\"endpoint_id\\\", \\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\") VALUES (:icinga2_event_handlers_enabled,:id,:icinga2_active_service_checks_enabled,:icinga2_start_time,:icinga2_performance_data_enabled,:responsible,:icinga2_flap_detection_enabled,:environment_id,:heartbeat,:icinga2_active_host_checks_enabled,:endpoint_id,:icinga2_notifications_enabled,:icinga2_version) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\",\\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\"\": pq: could not serialize access due to read/write dependencies among transactions","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","_TRANSPORT":"journal","_SYSTEMD_UNIT":"icingadb.service","__REALTIME_TIMESTAMP":"1732023417498561","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d0279e;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274ad279e48;t=62744208aebc1;x=a6969332bcaa6948","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a"}
{"_SYSTEMD_SLICE":"system.slice","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_HOSTNAME":"icm02","_UID":"999","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","_GID":"995","_EXE":"/usr/sbin/icingadb","PRIORITY":"6","_RUNTIME_SCOPE":"system","_CAP_EFFECTIVE":"0","_COMM":"icingadb","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"heartbeat\\\", \\\"environment_id\\\", \\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\", \\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\", \\\"id\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\") VALUES (:responsible,:icinga2_start_time,:icinga2_performance_data_enabled,:icinga2_flap_detection_enabled,:heartbeat,:environment_id,:endpoint_id,:icinga2_active_host_checks_enabled,:icinga2_notifications_enabled,:icinga2_version,:id,:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\",\\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\",\\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\"\": pq: could not serialize access due to read/write dependencies among transactions","SYSLOG_IDENTIFIER":"icingadb","_SOURCE_REALTIME_TIMESTAMP":"1732023420302546","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_TRANSPORT":"journal","_PID":"4121746","__MONOTONIC_TIMESTAMP":"2700147320676","_SELINUX_CONTEXT":"unconfined\n","MESSAGE":"high-availability: Can't update or insert instance. Retrying","_SYSTEMD_UNIT":"icingadb.service","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d0279f;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274ad526764;t=6274420b5b4dd;x=22c117cbec4444db","__REALTIME_TIMESTAMP":"1732023420302557","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml"}
{"_SOURCE_REALTIME_TIMESTAMP":"1732023422991492","_GID":"995","_SELINUX_CONTEXT":"unconfined\n","PRIORITY":"6","__MONOTONIC_TIMESTAMP":"2700150009623","__REALTIME_TIMESTAMP":"1732023422991504","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_CAP_EFFECTIVE":"0","SYSLOG_IDENTIFIER":"icingadb","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d027a0;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274ad7b6f17;t=6274420debc90;x=b41bbcb9a5b0d3bc","_EXE":"/usr/sbin/icingadb","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"id\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"heartbeat\\\", \\\"environment_id\\\", \\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\", \\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\") VALUES (:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled,:id,:responsible,:icinga2_start_time,:icinga2_performance_data_enabled,:icinga2_flap_detection_enabled,:heartbeat,:environment_id,:endpoint_id,:icinga2_active_host_checks_enabled,:icinga2_notifications_enabled,:icinga2_version) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\",\\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\",\\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\"\": pq: could not serialize access due to read/write dependencies among transactions","_RUNTIME_SCOPE":"system","_PID":"4121746","_COMM":"icingadb","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","_SYSTEMD_UNIT":"icingadb.service","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","_SYSTEMD_SLICE":"system.slice","_HOSTNAME":"icm02","MESSAGE":"high-availability: Can't update or insert instance. Retrying","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_TRANSPORT":"journal","_UID":"999"}
{"__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d027a2;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274ad9722d8;t=6274420fa7051;x=8575633620cb031c","_UID":"999","_COMM":"icingadb","__MONOTONIC_TIMESTAMP":"2700151825112","_SELINUX_CONTEXT":"unconfined\n","_SYSTEMD_SLICE":"system.slice","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"heartbeat\\\", \\\"environment_id\\\", \\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\", \\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"id\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"icinga2_flap_detection_enabled\\\") VALUES (:heartbeat,:environment_id,:endpoint_id,:icinga2_active_host_checks_enabled,:icinga2_notifications_enabled,:icinga2_version,:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled,:id,:responsible,:icinga2_start_time,:icinga2_performance_data_enabled,:icinga2_flap_detection_enabled) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\",\\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\",\\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\"\": pq: could not serialize access due to read/write dependencies among transactions","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","_PID":"4121746","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","_HOSTNAME":"icm02","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","MESSAGE":"high-availability: Can't update or insert instance. Retrying","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_SYSTEMD_UNIT":"icingadb.service","_GID":"995","__REALTIME_TIMESTAMP":"1732023424806993","_EXE":"/usr/sbin/icingadb","SYSLOG_IDENTIFIER":"icingadb","_TRANSPORT":"journal","_CAP_EFFECTIVE":"0","_RUNTIME_SCOPE":"system","_SOURCE_REALTIME_TIMESTAMP":"1732023424806975","PRIORITY":"6"}
{"_UID":"999","SYSLOG_IDENTIFIER":"icingadb","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","_PID":"4121746","_CMDLINE":"/usr/sbin/icingadb --config /etc/icingadb/config.yml","_COMM":"icingadb","MESSAGE":"high-availability: Can't update or insert instance. Retrying","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d027a4;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274adbd2913;t=627442120768d;x=2a8beef3accbda36","_HOSTNAME":"icm02","_SOURCE_REALTIME_TIMESTAMP":"1732023427298946","ICINGADB_ERROR":"can't perform \"INSERT INTO \\\"icingadb_instance\\\" (\\\"icinga2_notifications_enabled\\\", \\\"icinga2_version\\\", \\\"id\\\", \\\"icinga2_active_service_checks_enabled\\\", \\\"icinga2_event_handlers_enabled\\\", \\\"responsible\\\", \\\"icinga2_start_time\\\", \\\"icinga2_performance_data_enabled\\\", \\\"icinga2_flap_detection_enabled\\\", \\\"heartbeat\\\", \\\"environment_id\\\", \\\"endpoint_id\\\", \\\"icinga2_active_host_checks_enabled\\\") VALUES (:icinga2_notifications_enabled,:icinga2_version,:id,:icinga2_active_service_checks_enabled,:icinga2_event_handlers_enabled,:responsible,:icinga2_start_time,:icinga2_performance_data_enabled,:icinga2_flap_detection_enabled,:heartbeat,:environment_id,:endpoint_id,:icinga2_active_host_checks_enabled) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \\\"icinga2_notifications_enabled\\\" = EXCLUDED.\\\"icinga2_notifications_enabled\\\",\\\"icinga2_version\\\" = EXCLUDED.\\\"icinga2_version\\\",\\\"id\\\" = EXCLUDED.\\\"id\\\",\\\"icinga2_active_service_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_service_checks_enabled\\\",\\\"icinga2_event_handlers_enabled\\\" = EXCLUDED.\\\"icinga2_event_handlers_enabled\\\",\\\"responsible\\\" = EXCLUDED.\\\"responsible\\\",\\\"icinga2_start_time\\\" = EXCLUDED.\\\"icinga2_start_time\\\",\\\"icinga2_performance_data_enabled\\\" = EXCLUDED.\\\"icinga2_performance_data_enabled\\\",\\\"icinga2_flap_detection_enabled\\\" = EXCLUDED.\\\"icinga2_flap_detection_enabled\\\",\\\"heartbeat\\\" = EXCLUDED.\\\"heartbeat\\\",\\\"environment_id\\\" = EXCLUDED.\\\"environment_id\\\",\\\"endpoint_id\\\" = EXCLUDED.\\\"endpoint_id\\\",\\\"icinga2_active_host_checks_enabled\\\" = EXCLUDED.\\\"icinga2_active_host_checks_enabled\\\"\": pq: could not serialize access due to read/write dependencies among transactions","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","PRIORITY":"6","_TRANSPORT":"journal","_SYSTEMD_UNIT":"icingadb.service","_SYSTEMD_SLICE":"system.slice","__MONOTONIC_TIMESTAMP":"2700154317075","_EXE":"/usr/sbin/icingadb","_SELINUX_CONTEXT":"unconfined\n","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_CAP_EFFECTIVE":"0","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_RUNTIME_SCOPE":"system","_GID":"995","__REALTIME_TIMESTAMP":"1732023427298957"}
{"_TRANSPORT":"journal","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","PRIORITY":"2","_SYSTEMD_UNIT":"icingadb.service","__MONOTONIC_TIMESTAMP":"2700183732964","_COMM":"icingadb","_SELINUX_CONTEXT":"unconfined\n","_SOURCE_REALTIME_TIMESTAMP":"1732023456714799","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d027a6;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274af7e02e4;t=6274422e1505d;x=bf5264f5739c0180","_PID":"4121746","_CAP_EFFECTIVE":"0","_SYSTEMD_SLICE":"system.slice","_SYSTEMD_CGROUP":"/system.slice/icingadb.service","SYSLOG_IDENTIFIER":"icingadb","_HOSTNAME":"icm02","__REALTIME_TIMESTAMP":"1732023456714845","MESSAGE":"retry deadline exceeded\ngithub.com/icinga/icingadb/pkg/icingadb.(*HA).controller\n\tgithub.com/icinga/icingadb/pkg/icingadb/ha.go:163\nruntime.goexit\n\truntime/asm_amd64.s:1695\nHA aborted\ngithub.com/icinga/icingadb/pkg/icingadb.(*HA).abort.func1\n\tgithub.com/icinga/icingadb/pkg/icingadb/ha.go:131\nsync.(*Once).doSlow\n\tsync/once.go:74\nsync.(*Once).Do\n\tsync/once.go:65\ngithub.com/icinga/icingadb/pkg/icingadb.(*HA).abort\n\tgithub.com/icinga/icingadb/pkg/icingadb/ha.go:129\ngithub.com/icinga/icingadb/pkg/icingadb.(*HA).controller\n\tgithub.com/icinga/icingadb/pkg/icingadb/ha.go:163\nruntime.goexit\n\truntime/asm_amd64.s:1695\nHA exited with an error\nmain.run\n\tgithub.com/icinga/icingadb/cmd/icingadb/main.go:335\nmain.main\n\tgithub.com/icinga/icingadb/cmd/icingadb/main.go:37\nruntime.main\n\truntime/proc.go:271\nruntime.goexit\n\truntime/asm_amd64.s:1695","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","_SYSTEMD_INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_GID":"995","_RUNTIME_SCOPE":"system","_UID":"999"}
{"_SYSTEMD_SLICE":"-.slice","CODE_FILE":"src/core/unit.c","_GID":"0","__REALTIME_TIMESTAMP":"1732023456717075","_CMDLINE":"/lib/systemd/systemd --system --deserialize=28","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d027a7;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274af7e0b99;t=6274422e15913;x=34d0ff2a2b199395","_SYSTEMD_CGROUP":"/init.scope","_COMM":"systemd","CODE_LINE":"5671","_SOURCE_REALTIME_TIMESTAMP":"1732023456717032","CODE_FUNC":"unit_log_process_exit","UNIT":"icingadb.service","_SYSTEMD_UNIT":"init.scope","EXIT_CODE":"exited","COMMAND":"ExecStart","MESSAGE":"icingadb.service: Main process exited, code=exited, status=1/FAILURE","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_PID":"1","INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","__MONOTONIC_TIMESTAMP":"2700183735193","MESSAGE_ID":"98e322203f7a4ed290d09fe03c09fe15","SYSLOG_IDENTIFIER":"systemd","_RUNTIME_SCOPE":"system","_SELINUX_CONTEXT":"unconfined\n","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","PRIORITY":"5","SYSLOG_FACILITY":"3","_HOSTNAME":"icm02","_CAP_EFFECTIVE":"1ffffffffff","TID":"1","_TRANSPORT":"journal","_UID":"0","_EXE":"/usr/lib/systemd/systemd","EXIT_STATUS":"1"}
{"__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d027a8;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274af7e7a07;t=6274422e1c780;x=85c7a0fbd1210a7c","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_UID":"0","INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_SYSTEMD_CGROUP":"/init.scope","MESSAGE":"icingadb.service: Failed with result 'exit-code'.","_HOSTNAME":"icm02","_SYSTEMD_SLICE":"-.slice","_TRANSPORT":"journal","TID":"1","_RUNTIME_SCOPE":"system","_CMDLINE":"/lib/systemd/systemd --system --deserialize=28","MESSAGE_ID":"d9b373ed55a64feb8242e02dbe79a49c","_COMM":"systemd","CODE_FILE":"src/core/unit.c","SYSLOG_IDENTIFIER":"systemd","_CAP_EFFECTIVE":"1ffffffffff","_EXE":"/usr/lib/systemd/systemd","_PID":"1","CODE_FUNC":"unit_log_failure","_SELINUX_CONTEXT":"unconfined\n","_GID":"0","_SOURCE_REALTIME_TIMESTAMP":"1732023456717196","PRIORITY":"4","CODE_LINE":"5629","UNIT_RESULT":"exit-code","UNIT":"icingadb.service","SYSLOG_FACILITY":"3","__MONOTONIC_TIMESTAMP":"2700183763463","__REALTIME_TIMESTAMP":"1732023456745344","_SYSTEMD_UNIT":"init.scope","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67"}
{"_CAP_EFFECTIVE":"1ffffffffff","MESSAGE_ID":"ae8f7b866b0347b9af31fe1c80b127c0","CODE_FILE":"src/core/unit.c","_EXE":"/usr/lib/systemd/systemd","SYSLOG_IDENTIFIER":"systemd","_TRANSPORT":"journal","_BOOT_ID":"2b78aaa316c641d4a0c3e6cedd490e67","INVOCATION_ID":"216444df88074c05aa62605bd4f6a63a","_SYSTEMD_SLICE":"-.slice","MESSAGE":"icingadb.service: Consumed 1min 45.498s CPU time.","CPU_USAGE_NSEC":"105498396000","CODE_FUNC":"unit_log_resources","_GID":"0","__REALTIME_TIMESTAMP":"1732023456745454","PRIORITY":"6","_PID":"1","_RUNTIME_SCOPE":"system","_HOSTNAME":"icm02","__MONOTONIC_TIMESTAMP":"2700183763573","_COMM":"systemd","_MACHINE_ID":"1c90e2315e5e488a84b9dc0bde7f1550","_SYSTEMD_UNIT":"init.scope","_SELINUX_CONTEXT":"unconfined\n","UNIT":"icingadb.service","SYSLOG_FACILITY":"3","TID":"1","_CMDLINE":"/lib/systemd/systemd --system --deserialize=28","__CURSOR":"s=eb3afcdb5c194068ae4378c3d1c18b84;i=2d027a9;b=2b78aaa316c641d4a0c3e6cedd490e67;m=274af7e7a75;t=6274422e1c7ee;x=4328f42046e73eb3","_SYSTEMD_CGROUP":"/init.scope","CODE_LINE":"2503","_SOURCE_REALTIME_TIMESTAMP":"1732023456717440","_UID":"0"}

deric avatar Nov 19 '24 13:11 deric