OMERO.tables does not always retry if it fails to connect on startup

[manics]

This was seen on outreach (@pwalczysko)

Example of expected behaviour: Failed to get session on attempt 1, but suceeded after a retry (total 30s).

2020-01-15 19:08:41,132 INFO  [                       omero.util.Server] (MainThread) ********************************************************************************
2020-01-15 19:08:41,132 INFO  [                       omero.util.Server] (MainThread) Waiting 10000 ms on startup
2020-01-15 19:08:51,132 INFO  [                       omero.util.Server] (MainThread) Starting
2020-01-15 19:08:51,132 INFO  [                       omero.util.Server] (MainThread) Loaded dependency numpy (1.18.1)
2020-01-15 19:08:51,133 INFO  [                       omero.util.Server] (MainThread) Loaded dependency tables (3.6.1, hdf=error)
2020-01-15 19:08:51,134 INFO  [                    omero.util.Resources] (Thread-2  ) Starting
2020-01-15 19:09:01,839 INFO  [                             omero.utils] (MainThread) Failed to get session on attempt 1
2020-01-15 19:09:11,896 INFO  [                    omero.tables.TablesI] (MainThread) Using storage factory: omero.hdfstorageV2.HdfList
2020-01-15 19:09:11,985 INFO  [                       omero.util.Server] (MainThread) Entering main loop

Example of bug: Fails and exits instead of retrying (total 26s).

2020-01-17 20:53:43,281 INFO  [                       omero.util.Server] (MainThread) Waiting 10000 ms on startup
2020-01-17 20:53:53,281 INFO  [                       omero.util.Server] (MainThread) Starting
2020-01-17 20:53:53,281 INFO  [                       omero.util.Server] (MainThread) Loaded dependency numpy (1.18.1)
2020-01-17 20:53:53,281 INFO  [                       omero.util.Server] (MainThread) Loaded dependency tables (3.6.1, hdf=error)
2020-01-17 20:53:53,283 INFO  [                    omero.util.Resources] (Thread-2  ) Starting
2020-01-17 20:53:53,450 INFO  [                    omero.tables.TablesI] (MainThread) Using storage factory: omero.hdfstorageV2.HdfList
2020-01-17 20:53:53,970 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: Ice.ConnectionRefusedException:
Connection refused
2020-01-17 20:54:09,000 ERROR [                       omero.util.Server] (MainThread) Failed initialization
Traceback (most recent call last):
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/util/__init__.py", line 499, in run
    self.impl = self.impl_class(ctx)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/tables.py", line 449, in __init__
    raise exc
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/tables.py", line 436, in __init__
    self._get_repo()
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/tables.py", line 506, in _get_repo
    self.repo_mgr = self._internal_repo_cast(self.repo_mgr)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_Repositories_ice.py", line 2165, in checkedCast
    return _M_omero.grid.InternalRepositoryPrx.ice_checkedCast(proxy, '::omero::grid::InternalRepository', facetOrCtx, _ctx)
Ice.ConnectionRefusedException: Ice.ConnectionRefusedException:
Connection refused
2020-01-17 20:54:09,001 INFO  [                       omero.util.Server] (MainThread) Cleanup
2020-01-17 20:54:09,001 INFO  [                       omero.util.Server] (MainThread) Stopped
2020-01-17 20:54:09,001 INFO  [                       omero.util.Server] (MainThread) ********************************************************************************
2020-01-17 20:54:09,002 WARNI [                                  stderr] (MainThread) 2020-01-17 20:54:09.002356 Tables-0: error: Traceback (most recent call last):
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/util/__init__.py", line 499, in run
    self.impl = self.impl_class(ctx)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/tables.py", line 449, in __init__
    raise exc
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/tables.py", line 436, in __init__
    self._get_repo()
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/tables.py", line 506, in _get_repo
    self.repo_mgr = self._internal_repo_cast(self.repo_mgr)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_Repositories_ice.py", line 2165, in checkedCast
    return _M_omero.grid.InternalRepositoryPrx.ice_checkedCast(proxy, '::omero::grid::InternalRepository', facetOrCtx, _ctx)
Ice.ConnectionRefusedException: Ice.ConnectionRefusedException:
Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/Ice.py", line 1218, in doMain
    status = self.run(args)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/util/__init__.py", line 503, in run
    sys.exit(100)
SystemExit: 100
2020-01-17 20:54:09,002 INFO  [                    omero.util.Resources] (Thread-2  ) Halted
2020-01-17 20:54:09,006 WARNI [                          StreamRedirect] (MainThread) No attribute: closed
2020-01-17 20:54:09,006 WARNI [                          StreamRedirect] (MainThread) No attribute: closed
2020-01-17 20:54:09,037 INFO  [                    omero.tables.TablesI] (MainThread) Cleaning up
2020-01-17 20:54:09,037 INFO  [                    omero.tables.TablesI] (MainThread) Done

If startup fails Tables can be restarted manually: /opt/omero/server/OMERO.server/bin/omero admin ice server enable Tables-0