nautobot-app-device-onboarding icon indicating copy to clipboard operation
nautobot-app-device-onboarding copied to clipboard
verified publisher icon nautobot

device onboarding not releasing completed job connections to postgresql db

Open chick-tiger opened this issue 7 months ago • 7 comments

Environment

  • Nautobot version (Docker tag too if applicable):2.4.5
  • Python version: 3.11.8
  • Database platform, version: postgresql, 12.17
  • Middleware(s):

Steps to Reproduce

  1. before running any job, on db server, run
netstat -tpn |grep 5432
systemctl status postgresql.service

to capture current tcp:5432 sessions and any idle nautobot db connections

  1. run a few sync network data jobs.

3.check the tcp:5432 connections and nautobot db idle sessions. and login the db and perform below query to get all idle sessions and when the sessions were built

SELECT
    pid,
    usename as username,
    datname as database,
    client_addr as client_ip,
    state,
    EXTRACT(EPOCH FROM (now() - state_change))::integer as idle_seconds,
    query
FROM pg_stat_activity
WHERE state = 'idle'
  AND EXTRACT(EPOCH FROM (now() - state_change)) > 60  -- More than 1 minute idle
ORDER BY idle_seconds DESC;"

Expected Behavior

Expect not that many idle tcp sessions to be established, and they should be terminated after the jobs are completed, or at least it respects the 300 seconds db timeout setting in nautobot.

Observed Behavior

tcp sessions are never closed, and more and more tcp sessions are established between nautobot front and nautobot db, which eventually eat up all available sockets and now new job can be performed. below is an example error

nms-nbback-1:~]# systemctl status postgresql
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2024-06-26 10:04:58 PDT; 10 months 2 days ago
 Main PID: 2754685 (postmaster)
    Tasks: 105 (limit: 408554)
   Memory: 1.0G
   CGroup: /system.slice/postgresql.service
           ├─1425680 postgres: nautobot nautobot 10.1.1.202(51930) idle
           ├─1425681 postgres: nautobot nautobot 10.1.1.202(51938) idle
           ├─1425702 postgres: nautobot nautobot 10.1.1.202(52014) idle
           ├─1426983 postgres: nautobot nautobot 10.1.1.202(52394) idle
           ├─1426986 postgres: nautobot nautobot 10.1.1.202(53864) idle
           ├─1427105 postgres: nautobot nautobot 10.1.1.202(37812) idle
           ├─1427111 postgres: nautobot nautobot 10.1.1.202(37872) idle
           ├─1454209 postgres: nautobot nautobot 10.1.1.202(50442) idle
           ├─1454216 postgres: nautobot nautobot 10.1.1.202(50508) idle
           ├─1454217 postgres: nautobot nautobot 10.1.1.202(50518) idle
           ├─1454218 postgres: nautobot nautobot 10.1.1.202(50534) idle
           ├─1454223 postgres: nautobot nautobot 10.1.1.202(48464) idle
           ├─1454226 postgres: nautobot nautobot 10.1.1.202(48502) idle
           ├─1454228 postgres: nautobot nautobot 10.1.1.202(48518) idle
           ├─1454229 postgres: nautobot nautobot 10.1.1.202(48534) idle
           ├─1454230 postgres: nautobot nautobot 10.1.1.202(48548) idle
           ├─1454231 postgres: nautobot nautobot 10.1.1.202(48564) idle
           ├─1454232 postgres: nautobot nautobot 10.1.1.202(48570) idle
           ├─1454233 postgres: nautobot nautobot 10.1.1.202(48584) idle
           ├─1454235 postgres: nautobot nautobot 10.1.1.202(48606) idle
           ├─1454236 postgres: nautobot nautobot 10.1.1.202(48608) idle
           ├─1454237 postgres: nautobot nautobot 10.1.1.202(48614) idle
           ├─1454238 postgres: nautobot nautobot 10.1.1.202(48628) idle
...omit..

Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]: Commands to run: ['show inventory', 'show interface', 'show ip interface vrf all', 'show port-channel summary', 'show vrf interface']
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]: [2025-04-29 05:45:24,441: DEBUG/ForkPoolWorker-15] Commands to run: ['show inventory', 'show interface', 'show ip interface vrf all', 'show port-channel summary', 'show vrf interface']
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]: [2025-04-29 05:45:24,448: ERROR/ForkPoolWorker-15] Host 'p23-adhr1-acc-1': task 'netmiko_send_commands' failed with traceback:
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]: Traceback (most recent call last):
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:   File "/opt/nautobot/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:     self.connect()
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:   File "/opt/nautobot/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:     return func(*args, **kwargs)
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:            ^^^^^^^^^^^^^^^^^^^^^
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:   File "/opt/nautobot/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:     self.connection = self.get_new_connection(conn_params)
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:   File "/opt/nautobot/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:     return func(*args, **kwargs)
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:            ^^^^^^^^^^^^^^^^^^^^^
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:   File "/opt/nautobot/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:     connection = self.Database.connect(**conn_params)
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:   File "/opt/nautobot/lib/python3.11/site-packages/psycopg2/__init__.py", line 122, in connect
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 28 22:45:24 nms-nbfront-1 nautobot-server[3784091]: psycopg2.OperationalError: connection to server at "10.1.1.203", port 5432 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections


nautobot=# SELECT 
nautobot-#     pid,
nautobot-#     usename as username,
nautobot-#     datname as database,
nautobot-#     client_addr as client_ip,
nautobot-#     state,
nautobot-#     EXTRACT(EPOCH FROM (now() - state_change))::integer as idle_seconds,
nautobot-#     query
nautobot-# FROM pg_stat_activity 
nautobot-# WHERE state = 'idle'
nautobot-#   AND EXTRACT(EPOCH FROM (now() - state_change)) > 60  -- More than 1 minute idle
nautobot-# ORDER BY idle_seconds DESC;"
   pid   | username | database |   client_ip   | state | idle_seconds |                                                                                                                                                                                                                                                                                                                                                                                                         
                                                                                                                     query                                                                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                        
---------+----------+----------+---------------+-------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 1613811 | nautobot | nautobot | 10.1.1.202 | idle  |        58997 | INSERT INTO "extras_joblogentry" ("id", "job_result_id", "log_level", "grouping", "message", "created", "log_object", "absolute_url") VALUES ('3aed370f-df78-4c55-a3a7-0f3e5e8e325c'::uuid, 'f0b31c3a-19e3-4fd8-8c31-a52a9fa1850e'::uuid, 'info', 'main', 'Task instance completed. Task Name: netmiko_send_commands', '2025-04-29T06:08:00.599334+00:00'::timestamptz, '', '')
 1613826 | nautobot | nautobot | 10.1.1.202 | idle  |        58997 | SELECT COUNT(*) AS "__count" FROM "extras_gitrepository" WHERE "extras_gitrepository"."provided_contents" @> '"nautobot_device_onboarding.onboarding_command_mappers"'
 1623131 | nautobot | nautobot | 10.1.1.202 | idle  |        56824 | SELECT COUNT(*) AS "__count" FROM "extras_gitrepository" WHERE "extras_gitrepository"."provided_contents" @> '"nautobot_device_onboarding.onboarding_command_mappers"'
 1623122 | nautobot | nautobot | 10.1.1.202 | idle  |        56824 | INSERT INTO "extras_joblogentry" ("id", "job_result_id", "log_level", "grouping", "message", "created", "log_object", "absolute_url") VALUES ('97386558-ba68-419b-aa8a-486f58e7ea48'::uuid, '1b6d3f82-ed95-45a8-b61c-37813bb8208a'::uuid, 'info', 'main', 'Task instance completed. Task Name: netmiko_send_commands', '2025-04-29T06:44:13.821413+00:00'::timestamptz, '', '')
 1623129 | nautobot | nautobot | 10.1.1.202 | idle  |        56824 | SELE

chick-tiger avatar Apr 29 '25 22:04 chick-tiger 

nautobot=# SELECT 
nautobot-#     pid,
nautobot-#     usename as username,
nautobot-#     datname as database,
nautobot-#     client_addr as client_ip,
nautobot-#     state,
nautobot-#     EXTRACT(EPOCH FROM (now() - state_change))::integer as idle_seconds,
nautobot-#     query
nautobot-# FROM pg_stat_activity 
nautobot-# WHERE state = 'idle'
nautobot-#   AND EXTRACT(EPOCH FROM (now() - state_change)) > 60  -- More than 1 minute idle
nautobot-# ORDER BY idle_seconds DESC;"
   pid   | username | database |   client_ip   | state | idle_seconds |                                                                                                                                                                                                                                                                                                                                                                                                         
                                                                                                                     query                                                                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                        
---------+----------+----------+---------------+-------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 1613811 | nautobot | nautobot | 10.1.1.202 | idle  |        58997 | INSERT INTO "extras_joblogentry" ("id", "job_result_id", "log_level", "grouping", "message", "created", "log_object", "absolute_url") VALUES ('3aed370f-df78-4c55-a3a7-0f3e5e8e325c'::uuid, 'f0b31c3a-19e3-4fd8-8c31-a52a9fa1850e'::uuid, 'info', 'main', 'Task instance completed. Task Name: netmiko_send_commands', '2025-04-29T06:08:00.599334+00:00'::timestamptz, '', '')
 1613826 | nautobot | nautobot | 10.1.1.202 | idle  |        58997 | SELECT COUNT(*) AS "__count" FROM "extras_gitrepository" WHERE "extras_gitrepository"."provided_contents" @> '"nautobot_device_onboarding.onboarding_command_mappers"'
 1623131 | nautobot | nautobot | 10.1.1.202 | idle  |        56824 | SELECT COUNT(*) AS "__count" FROM "extras_gitrepository" WHERE "extras_gitrepository"."provided_contents" @> '"nautobot_device_onboarding.onboarding_command_mappers"'
 1623122 | nautobot | nautobot | 10.1.1.202 | idle  |        56824 | INSERT INTO "extras_joblogentry" ("id", "job_result_id", "log_level", "grouping", "message", "created", "log_object", "absolute_url") VALUES ('97386558-ba68-419b-aa8a-486f58e7ea48'::uuid, '1b6d3f82-ed95-45a8-b61c-37813bb8208a'::uuid, 'info', 'main', 'Task instance completed. Task Name: netmiko_send_commands', '2025-04-29T06:44:13.821413+00:00'::timestamptz, '', '')
 1623129 | nautobot | nautobot | 10.1.1.202 | idle  |        56824 | SELECT COUNT(*) AS "__count" FROM "extras_gitrepository" WHERE "extras_gitrepository"."provided_contents" @> '"nautobot_device_onboarding.onboarding_command_mappers"'
 1623123 | nautobot | nautobot | 10.1.1.202 | idle  |        56824 | INSERT INTO "extras_joblogentry" ("id", "job_result_id", "log_level", "grouping", "message", "created", "log_object", "absolute_url") VALUES ('3b72becb-8432-4472-a763-199242e13454'::uuid, '2ba30219-e7a8-4c6c-b62c-469a0b345b1a'::uuid, 'info', 'main', 'Task instance completed. Task Name: netmiko_send_commands', '2025-04-29T06:44:13.826954+00:00'::timestamptz, '', '')
 1623130 | nautobot | nautobot | 10.1.1.202 | idle  |        56823 | INSERT INTO "extras_joblogentry" ("id", "job_result_id", "log_level", "grouping", "message", "created", "log_object", "absolute_url") VALUES ('d9a72406-694b-456c-be28-414abe107080'::uuid, '7a2cfd03-0200-4654-a596-56235586c789'::uuid, 'info', 'main', 'Task instance completed. Task Name: netmiko_send_commands', '2025-04-29T06:44:14.980776+00:00'::timestamptz, '', '')
 1623133 | nautobot | nautobot | 10.1.1.202 | idle  |        56823 | SELECT COUNT(*) AS "__count" FROM "extras_gitrepository" WHERE "extras_gitrepository"."provided_contents" @> '"nautobot_device_onboarding.onboarding_command_mappers"'
 1623119 | nautobot | nautobot | 10.1.1.202 | idle  |        56822 | INSERT INTO "extras_joblogentry" ("id", "job_result_id", "log_level", "grouping", "message", "created", "log_object", "absolute_url") VALUES ('60cb1176-de86-4474-8d55-c11310a46a12'::uuid, '2ba30219-e7a8-4c6c-b62c-469a0b345b1a'::uuid, 'success', 'post_run', 'Job completed', '2025-04-29T06:44:15.140708+00:00'::timestamptz, '', '')
 1623117 | nautobot | nautobot | 10.1.1.202 | idle  |        56822 | INSERT INTO "extras_joblogentry" ("id", "job_result_id", "log_level", "grouping", "message", "created", "log_object", "absolute_url") VALUES ('60b3405b-207c-4b2d-b0f7-300e7fe1fd34'::uuid, '1b6d3f82-ed95-45a8-b61c-37813bb8208a'::uuid, 'success', 'post_run', 'Job completed', '2025-04-29T06:44:15.374278+00:00'::timestamptz, '', '')
 1623116 | nautobot | nautobot | 10.1.1.202 | idle  |        56822 | UPDATE "extras_jobresult" SET "_custom_field_data" = '{}', "job_model_id" = '8cb11d7e-70e8-4ce4-86e8-9e068aa0c2c6'::uuid, "name" = 'Sync Network Data From Network', "task_name" = 'nautobot_device_onboarding.jobs.SSOTSyncNetworkData', "date_created" = '2025-04-29T06:44:11.219273+00:00'::timestamptz, "date_done" = '2025-04-29T06:44:15.377699+00:00'::timestamptz, "user_id" = '27980429-ffdd-47
22-a8d3-4e5e0d91fe12'::uuid, "status" = 'SUCCESS', "result" = NULL, "worker" = 'celery@p23-nms-nbfront-1', "task_args" = '[]', "task_kwargs" = '{"dryrun": false, "memory_profiling": false, "debug": false, "connectivity_test": false, "sync_vlans": false, "sync_vrfs": true, "sync_cables": false, "namespace": "e0131c75-e6c5-4804-b6c1-5674a453b2c2", "interface_status": "176bb060-2783-4d3b-b988-115dbc736357", "ip_address_status": "176bb060-2783-4d3b-b988-115dbc7363
57", "default_prefix_status": "176bb060-2783-4d3b-b988-115dbc736357", "devices": ["f81b4c8b-cd66-4766-924e-5ff845c914a8"], "location": null, "device_role": null, "plat
 1623132 | nautobot | nautobot | 10.1.1.202 | idle  |        56822 | SELECT COUNT(*) AS "__count" FROM "extras_gitrepository" WHERE "extras_gitrepository"."provided_contents" @> '"nautobot_device_onboarding.onboarding_command_mappers"'
 1623124 | nautobot | nautobot | 10.1.1.202 | idle  |        56822 | INSERT INTO "extras_joblogentry" ("id", "job_result_id", "log_level", "grouping", "message", "created", "log_object", "absolute_url") VALUES ('a6ee0de9-a908-4686-8b08-6ba124edfbbe'::uuid, '8a2d7a7b-f83b-4ca9-a94c-ea16d4c1d895'::uuid, 'info', 'main', 'Task instance completed. Task Name: netmiko_send_commands', '2025-04-29T06:44:15.503204+00:00'::timestamptz, '', '')
 1623118 | nautobot | nautobot | 10.1.1.202 | idle  |        56822 | UPDATE "extras_jobresult" SET "_custom_field_data" = '{}', "job_model_id" = '8cb11d7e-70e8-4ce4-86e8-9e068aa0c2c6'::uuid, "name" = 'Sync Network Data From Network', "task_name" = 'nautobot_device_onboarding.jobs.SSOTSyncNetworkData', "date_created" = '2025-04-29T06:44:11.279740+00:00'::timestamptz, "date_done" = '2025-04-29T06:44:15.144689+00:00'::timestamptz, "user_id" = '27980429-ffdd-47
22-a8d3-4e5e0d91fe12'::uuid, "status" = 'SUCCESS', "result" = NULL, "worker" = 'celery@p23-nms-nbfront-1', "task_args" = '[]', "task_kwargs" = '{"dryrun": false, "memory_profiling": false, "debug": false, "connectivity_test": false, "sync_vlans": false, "sync_vrfs": true, "sync_cables": false, "namespace": "e0131c75-e6c5-4804-b6c1-5674a453b2c2", "interface_status": "176bb060-2783-4d3b-b988-115dbc736357", "ip_address_status": "176bb060-2783-4d3b-b988-115dbc7363
57", "default_prefix_status": "176bb060-2783-4d3b-b988-115dbc736357", "devices": ["624c6249-4d1a-474c-a190-1e63f076e340"], "location": null, "device_role": null, "plat
 1623120 | nautobot | nautobot | 10.1.1.202 | idle  |        56821 | UPDATE "extras_jobresult" SET "_custom_field_data" = '{}', "job_model_id" = '8cb11d7e-70e8-4ce4-86e8-9e068aa0c2c6'::uuid, "name" = 'Sync Network Data From Network', "task_name" = 'nautobot_device_onboarding.jobs.SSOTSyncNetworkData', "date_created" = '2025-04-29T06:44:11.323885+00:00'::timestamptz, "date_done" = '2025-04-29T06:44:16.842650+00:00'::timestamptz, "user_id" = '27980429-ffdd-47
22-a8d3-4e5e0d91fe12'::uuid, "status" = 'SUCCESS', "result" = NULL, "worker" = 'celery@p23-nms-nbfront-1', "task_args" = '[]', "task_kwargs" = '{"dryrun": false, "memory_profiling": false, "debug": false, "connectivity_test": false, "sync_vlans": false, "sync_vrfs": true, "sync_cables": false, "namespace": "e0131c75-e6c5-4804-b6c1-5674a453b2c2", "interface_status": "176bb060-2783-4d3b-b988-115dbc736357", "ip_address_status": "176bb060-2783-4d3b-b988-115dbc7363
57", "default_prefix_status": "176bb060-2783-4d3b-b988-115dbc736357", "devices": ["bb9efa43-b7e3-4ad8-b5f3-7496afbdb845"], "location": null, "device_role": null, "plat
 1623128 | nautobot | nautobot | 10.1.1.202 | idle  |        56821 | INSERT INTO "extras_joblogentry" ("id", "job_result_id", "log_level", "grouping", "message", "created", "log_object", "absolute_url") VALUES ('124b2930-08b5-4ed9-8f0a-6466e28273fd'::uuid, '7a2cfd03-0200-4654-a596-56235586c789'::uuid, 'success', 'post_run', 'Job completed', '2025-04-29T06:44:16.235895+00:00'::timestamptz, '', '')
 1623127 | nautobot | nautobot | 10.1.1.202 | idle  |        56821 | UPDATE "extras_jobresult" SET "_custom_field_data" = '{}', "job_model_id" = '8cb11d7e-70e8-4ce4-86e8-9e068aa0c2c6'::uuid, "name" = 'Sync Network Data From Network', "task_name" = 'nautobot_device_onboarding.jobs.SSOTSyncNetworkData', "date_created" = '2025-04-29T06:44:12.445110+00:00'::timestamptz, "date_done" = '2025-04-29T06:44:16.239430+00:00'::timestamptz, "user_id" = '27980429-ffdd-47
22-a8d3-4e5e0d91fe12'::uuid, "status" = 'SUCCESS', "result" = NULL, "worker" = 'celery@p23-nms-nbfront-1', "task_args" = '[]', "task_kwargs" = '{"dryrun": false, "memory_profiling": false, "debug": false, "connectivity_test": false, "sync_vlans": false, "sync_vrfs": true, "sync_cables": false, "namespace": "e0131c75-e6c5-4804-b6c1-5674a453b2c2", "interface_status": "176bb060-2783-4d3b-b988-115dbc736357", "ip_address_status": "176bb060-2783-4d3b-b988-115dbc7363
57", "default_prefix_status": "176bb060-2783-4d3b-b988-115dbc736357", "devices": ["756c7987-1eb6-476d-bb2b-119588d8fb13"], "location": null, "device_role": null, "plat
 1623126 | nautobot | nautobot | 10.1.1.202 | idle  |        56821 | SELECT COUNT(*) AS "__count" FROM "extras_gitrepository" WHERE "extras_gitrepository"."provided_contents" @> '"nautobot_device_onboarding.onboarding_command_mappers"'
 1623121 | nautobot | nautobot | 10.1.1.202 | idle  |        56821 | INSERT INTO "extras_joblogentry" ("id", "job_result_id", "log_level", "grouping", "message", "created", "log_object", "absolute_url") VALUES ('3a86ef9b-6a06-468f-8928-6ad6de18ed88'::uuid, '8a2d7a7b-f83b-4ca9-a94c-ea16d4c1d895'::uuid, 'success', 'post_run', 'Job completed', '2025-04-29T06:44:16.839523+00:00'::timestamptz, '', '')
 1623114 | nautobot | nautobot | 10.1.1.202 | idle  |        56821 | INSERT INTO "extras_joblogentry" ("id", "job_result_id", "log_level", "grouping", "message", "created", "log_object", "absolute_url") VALUES ('9559cb36-9d37-4ec7-9b1b-7c0797c25d2e'::uuid, '81f5f806-c063-4d9d-b41d-9a754c70d1a8'::uuid, 'info', 'main', 'Task instance completed. Task Name: netmiko_send_commands', '2025-04-29T06:44:16.205599+00:00'::timestamptz, '', '')
 1623109 | nautobot | nautobot | 10.1.1.202 | idle  |        56819 | INSERT INTO "extras_joblogentry" ("id", "job_result_id", "log_level", "grouping", "message", "created", "log_object", "absolute_url") VALUES ('71e91e2f-474c-464e-9984-6f5fb2b00faa'::uuid, '81f5f806-c063-4d9d-b41d-9a754c70d1a8'::uuid, 'success', 'post_run', 'Job completed', '2025-04-29T06:44:18.558232+00:00'::timestamptz, '', '')
nautobot"#

chick-tiger avatar Apr 29 '25 22:04 chick-tiger

Moved from the Nautobot repo and pining @jeffkala

lampwins avatar May 05 '25 14:05 lampwins

Thanks @lampwins and @chick-tiger, I believe this PR has a fix https://github.com/nautobot/nautobot-app-golden-config/pull/352

scetron avatar May 14 '25 15:05 scetron

@jeffkala @lampwins any plan to add similar fix as in nautobot-app-golden-config to device onboarding? Thanks.

chick-tiger avatar May 27 '25 16:05 chick-tiger

@chick-tiger yes, likely we will bring a similar fix over to DO, it is not yet scheduled for a release.

scetron avatar May 27 '25 18:05 scetron

I was digging on an issue for running out of connections (in the default pool of 100) for Device Onboarding App and found this public issue. I was previously aware of the patch for close_threaded_db_connections decorator from Golden Config referenced so my thought it was the same case.

Since this is currently in Open status I will say for others, one mitigation might be to raise MAX_CONNECTIONS in PG to 200 or 300. I recently was made aware of some other settings you might consider if you did raise that here.

MeganerdDev avatar Jul 15 '25 17:07 MeganerdDev

This will be worked on shortly (I don't have a definite ETA). We will likely use a similar fix as found with Golden Config. I think the longer term fix will be to implement a context manager for the functions that are opening DB connections.

scetron avatar Aug 01 '25 20:08 scetron