Handle issues with postgres more gracefully

[bobpaul]

Last night I restarted the postgresql service for reasons... and it resulted in mautrix crashing. It would probably be better if there was some retry logic to recover from this, as I imagine the same scenario could happen if the postgres database were on a remote system and there was a network failure.

This also relates to #12 as it would have been nice if users all got a message from the bot while it was shutting down/starting back up.

Feb 27 03:39:50 archlinux mautrix-hangouts[53697]: [2020-02-27 03:39:50,813] [[email protected]] Client.connect returning because Channel.listen returned
Feb 27 03:39:50 archlinux mautrix-hangouts[53697]: [2020-02-27 03:39:50,813] [[email protected].@REDACTED:REDACTED] Client connection finished
i
Feb 27 03:39:50 archlinux mautrix-hangouts[53697]: [2020-02-27 03:39:50,814] [[email protected]] Stopping puppet syncers
Feb 27 03:39:50 archlinux mautrix-hangouts[53697]: [2020-02-27 03:39:50,814] [[email protected]] Saving user sessions
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]: Traceback (most recent call last):
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     self.dialect.do_execute(
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     cursor.execute(statement, parameters)
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]: psycopg2.errors.AdminShutdown: terminating connection due to administrator command
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]: server closed the connection unexpectedly
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:         This probably means the server terminated abnormally
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:         before or while processing the request.
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]: The above exception was the direct cause of the following exception:
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]: Traceback (most recent call last):
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/bin/mautrix-hangouts", line 11, in <module>
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     load_entry_point('mautrix-hangouts==0.1.0.dev7+dev.1241db9b', 'console_scripts', 'mautrix-hangouts')()
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 490, in load_entry_point
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     return get_distribution(dist).load_entry_point(group, name)
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2854, in load_entry_point
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     return ep.load()
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2445, in load
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     return self.resolve()
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2451, in resolve
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     module = __import__(self.module_name, fromlist=['__name__'], level=0)
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/mautrix_hangouts/__main__.py", line 78, in <module>
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     HangoutsBridge().run()
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/mautrix/bridge/bridge.py", line 84, in run
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     self._run()
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/mautrix/bridge/bridge.py", line 210, in _run
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     self.prepare_shutdown()
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/mautrix_hangouts/__main__.py", line 75, in prepare_shutdown
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     user.save()
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/mautrix_hangouts/user.py", line 92, in save
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     self.db_instance.edit(refresh_token=self.refresh_token, gid=self.gid)
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/mautrix/util/db/base.py", line 157, in edit
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     conn.execute(self.t.update()
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 982, in execute
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     return meth(self, multiparams, params)
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     return connection._execute_clauseelement(self, multiparams, params)
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1095, in _execute_clauseelement
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     ret = self._execute_context(
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     self._handle_dbapi_exception(
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     util.raise_from_cause(sqlalchemy_exception, exc_info)
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     reraise(type(exception), exception, tb=exc_tb, cause=cause)
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     raise value.with_traceback(tb)
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     self.dialect.do_execute(
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:   File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:     cursor.execute(statement, parameters)
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]: sqlalchemy.exc.OperationalError: (psycopg2.errors.AdminShutdown) terminating connection due to administrator command
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]: server closed the connection unexpectedly
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:         This probably means the server terminated abnormally
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]:         before or while processing the request.
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]: [SQL: UPDATE "user" SET gid=%(gid)s, refresh_token=%(refresh_token)s WHERE "user".mxid = %(mxid_1)s]
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]: [parameters: {'gid': 'REDACTED', 'refresh_token': 'REDACTED', 'mxid_1': '@REDACTED:REDACTED'}]
Feb 27 03:39:51 archlinux mautrix-hangouts[53697]: (Background on this error at: http://sqlalche.me/e/e3q8)
Feb 27 03:39:52 archlinux systemd[1]: mautrix-hangouts.service: Main process exited, code=exited, status=1/FAILURE
Feb 27 03:39:52 archlinux systemd[1]: mautrix-hangouts.service: Failed with result 'exit-code'.
Feb 27 03:39:52 archlinux systemd[1]: Stopped Hangouts Matrix Puppet.