ERROR: could not obtain lock on relation "metric_temp_files_size_delta_current"

[wsescu]

Dear temBoard support team

I do get every 3h the following ERROR:

$ cat postgresql-Fri.log | grep ERROR
2022-05-20 00:43:27.195 CEST [21739] ERROR:  could not obtain lock on relation "metric_loadavg_current"
2022-05-20 00:43:28.124 CEST [21739] ERROR:  could not obtain lock on relation "metric_btree_bloat_current"
2022-05-20 03:37:01.606 CEST [13977] ERROR:  could not obtain lock on relation "metric_temp_files_size_delta_current"
2022-05-20 03:37:02.685 CEST [13977] ERROR:  could not obtain lock on relation "metric_loadavg_current"
2022-05-20 03:37:02.686 CEST [13977] ERROR:  could not obtain lock on relation "metric_filesystems_size_current"
2022-05-20 06:36:15.311 CEST [14326] ERROR:  could not obtain lock on relation "metric_temp_files_size_delta_current"
2022-05-20 06:36:18.047 CEST [14326] ERROR:  could not obtain lock on relation "metric_heap_bloat_current"

I am running temBoard version 7.11

$ temboard --version
 INFO: Starting temboard 7.11.
temBoard 7.11
System Red Hat Enterprise Linux Server 7.9 (Maipo)
Python 2.7.5 (/usr/bin/python)
Tornado 4.2.1
psycopg2 2.5.1 (dt dec pq3 ext)
libpq 9.2.24
SQLAlchemy 0.9.8
alembic 0.8.3

Would be cool, if you could help me to fix that issue.

Cheers William

[bersace]

Hi @wsescu , thanks for the report.

I think it's a duplicate of #1031 .

I guess that history_tables background task is locking _current tables for too long. Blocking collect of metrics. temBoard should know how to collect the missing metrics once history_tables has ended.

Do you see a hole in graph at these dates ?

[bersace]

@wsescu @bsislow can you confirm this is the same issue ? The logs are a bit different but looks very similar. If you're ok, let's continue the discussion on #1031.

[wsescu]

Dear @bersace

since the upgrade of temBoard to version 7.9 in January 2022, I do not see any history information for any instance. e.g. if I select a 1 year time frame.

Don't know if this is related to the locking issue.

Cheers, William

[bsislow]

Yes, this looks similar. If I look back a couple of weeks I see the issue intermittently in our Postgres logs:

postgres-2022-05-07_000000.log:2022-05-07 20:07:33 CDT [26555]: [1-1] user=temboard,db=temboard,app=temboard,client=[local] ERROR:  could not obtain lock on relation "metric_locks_current"
postgres-2022-05-07_000000.log:2022-05-07 20:07:36 CDT [26555]: [4-1] user=temboard,db=temboard,app=temboard,client=[local] ERROR:  could not obtain lock on relation "metric_bgwriter_current"
postgres-2022-05-11_000000.log:2022-05-11 17:08:09 CDT [18834]: [1-1] user=temboard,db=temboard,app=temboard,client=[local] ERROR:  could not obtain lock on relation "metric_memory_current"
postgres-2022-05-11_000000.log:2022-05-11 17:08:09 CDT [18834]: [4-1] user=temboard,db=temboard,app=temboard,client=[local] ERROR:  could not obtain lock on relation "metric_temp_files_size_delta_current"
postgres-2022-05-11_000000.log:2022-05-11 17:08:09 CDT [18834]: [7-1] user=temboard,db=temboard,app=temboard,client=[local] ERROR:  could not obtain lock on relation "metric_btree_bloat_current"
postgres-2022-05-11_000000.log:2022-05-11 20:08:09 CDT [23376]: [1-1] user=temboard,db=temboard,app=temboard,client=[local] ERROR:  could not obtain lock on relation "metric_db_size_current"
postgres-2022-05-14_000000.log:2022-05-14 08:08:33 CDT [20884]: [1-1] user=temboard,db=temboard,app=temboard,client=[local] ERROR:  could not obtain lock on relation "metric_cpu_current"
postgres-2022-05-14_000000.log:2022-05-14 11:08:29 CDT [25087]: [1-1] user=temboard,db=temboard,app=temboard,client=[local] ERROR:  could not obtain lock on relation "metric_locks_current"
postgres-2022-05-14_000000.log:2022-05-14 11:08:33 CDT [25087]: [4-1] user=temboard,db=temboard,app=temboard,client=[local] ERROR:  could not obtain lock on relation "metric_temp_files_size_delta_current"
postgres-2022-05-14_000000.log:2022-05-14 14:08:34 CDT [29381]: [1-1] user=temboard,db=temboard,app=temboard,client=[local] ERROR:  could not obtain lock on relation "metric_bgwriter_current"
postgres-2022-05-18_000000.log:2022-05-18 02:09:01 CDT [28565]: [1-1] user=temboard,db=temboard,app=temboard,client=[local] ERROR:  could not obtain lock on relation "metric_blocks_current"
postgres-2022-05-18_000000.log:2022-05-18 02:09:02 CDT [28565]: [4-1] user=temboard,db=temboard,app=temboard,client=[local] ERROR:  could not obtain lock on relation "metric_tblspc_size_current"
postgres-2022-05-18_000000.log:2022-05-18 05:09:04 CDT [770]: [1-1] user=temboard,db=temboard,app=temboard,client=[local] ERROR:  could not obtain lock on relation "metric_db_size_current"

[wsescu]

Since I moved to temBoard version 8.1, this issue is solved. Thanks.

Cheers William