MySQL -> Failed to Get Lock

[jvanderaa]

Environment

• Python version: 3.10
• Nautobot version: 1.4.1
• nautobot-golden-config version: 1.2

Steps to Reproduce

1. Move from Postgres to MySQL since I am still early in the data in this instance
2. Try to do config backup which worked on postgres
3. Failure below

Expected Behavior

Config backed up

Observed Behavior

Error:

  File "/opt/nautobot/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/opt/nautobot/lib/python3.10/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/opt/nautobot/lib/python3.10/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/opt/nautobot/lib/python3.10/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
MySQLdb.OperationalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_golden_config/nornir_plays/config_backup.py", line 133, in config_backup
    nr_with_processors.run(
  File "/opt/nautobot/lib/python3.10/site-packages/nornir/core/__init__.py", line 139, in run
    result = self.runner.run(task, run_on)
  File "/opt/nautobot/lib/python3.10/site-packages/nornir/plugins/runners/__init__.py", line 43, in run
    worker_result = future.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/nautobot/lib/python3.10/site-packages/nornir/core/task.py", line 136, in start
    self.processors.task_instance_completed(self, host, self.results)
  File "/opt/nautobot/lib/python3.10/site-packages/nornir/core/processor.py", line 85, in task_instance_completed
    p.task_instance_completed(task, host, result)
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_golden_config/nornir_plays/processor.py", line 34, in task_instance_completed
    self.logger.log_failure(task.host.data["obj"], f"{task.name} failed: {result.exception}")
  File "/opt/nautobot/lib/python3.10/site-packages/nornir_nautobot/utils/logger.py", line 49, in log_failure
    self.nautobot_job.log_failure(obj, message)
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot/extras/jobs.py", line 575, in log_failure
    self._log(obj, message, level_choice=LogLevelChoices.LOG_FAILURE)
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot/extras/jobs.py", line 517, in _log
    self.job_result.log(
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot/extras/models/jobs.py", line 760, in log
    log.save(using=JOB_LOGS)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/opt/nautobot/lib/python3.10/site-packages/cacheops/transaction.py", line 97, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
    with self.db.wrap_database_errors:
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/opt/nautobot/lib/python3.10/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/opt/nautobot/lib/python3.10/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/opt/nautobot/lib/python3.10/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/opt/nautobot/lib/python3.10/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/opt/nautobot/lib/python3.10/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/opt/nautobot/lib/python3.10/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
MySQLdb.OperationalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot/extras/jobs.py", line 1172, in _run_job
    output = job.run(data=data, commit=commit)
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_golden_config/jobs.py", line 51, in inner
    return method(obj, data, commit)
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_golden_config/jobs.py", line 189, in run
    config_backup(self, data)
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_golden_config/nornir_plays/config_backup.py", line 144, in config_backup
    logger.log_failure(None, err)
  File "/opt/nautobot/lib/python3.10/site-packages/nornir_nautobot/utils/logger.py", line 49, in log_failure
    self.nautobot_job.log_failure(obj, message)
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot/extras/jobs.py", line 575, in log_failure
    self._log(obj, message, level_choice=LogLevelChoices.LOG_FAILURE)
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot/extras/jobs.py", line 517, in _log
    self.job_result.log(
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot/extras/models/jobs.py", line 760, in log
    log.save(using=JOB_LOGS)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/opt/nautobot/lib/python3.10/site-packages/cacheops/transaction.py", line 97, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
    with self.db.wrap_database_errors:
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/nautobot/lib/python3.10/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/opt/nautobot/lib/python3.10/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/opt/nautobot/lib/python3.10/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/opt/nautobot/lib/python3.10/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')

[itdependsnetworks]

Can you test with @jeffkala? This is likely since we are not releasing the sql connection correctly, but looking for a good place to disconnect and not finding a good test env.

[milan2655]

I have the same problem. Is there any solution for this?