clp-package: Add connection pool option to SQL_Adapter; Make search scheduler resilient to database connection failures

[gibber9809]

Description

This PR makes several changes in sql_adapter.py and search_scheduler.py to improve resilience with database connection issues.

In SQL_Adapter we introduce an option to create a connection pool using sqlalchemy to handle the details of maintaining a pool of living database connections. Since the connect() method can throw we wrap the sqlalchemy connection pool with a class that returns a dummy connection on error, and logs the connection errors (with a mechanism to ensure connection errors are only logged periodically to avoid log spew).

The dummy connection implements the close() method which means all of the code using the connection pool can be written like

with contextlib.closing(db_conn_pool.connect()) as db_conn:
...

without fear of throwing on the initial connect() call.

In the search scheduler we move all database operations into functions wrapped with a new decorator that catches database exceptions, and returns some default value on database failure. This means that code interacting with the database can be happily ignorant of database failures when there are errors on a real database connection, and when a dummy database connection has been provided by the connection pool.

Validation performed

• Validated that the search scheduler continues functioning after bouncing mariadb database
• Validated that the search scheduler continues functioning after bouncing mysql database
• Validated that connection errors are only logged periodically to avoid log spew