napari-omero
napari-omero copied to clipboard
tlambert03
Connection Lost Exception: no clear way to reconnect
After having connected to an OMERO server and idled for some time, I lost connection -- likely because of computer sleep. When returning to napari, I get the Connection Lost Exception. The image I had loaded is fine. Getting an exception is fine, but there is no clear path to recovery -- the Disconnect button remains, but does't work, and there is no reconnect/login option.
What would be nice is if the Disconnect button became Reconnect, prompting for a password if needed, but otherwise returning the session.
---------------------------------------------------------------------------
ConnectionLostException Traceback (most recent call last)
File ~/Documents/dev/napari-omero/src/napari_omero/widgets/tree_model.py:94, in OMEROTreeModel.fetchMore(self=<napari_omero.widgets.tree_model.OMEROTreeModel object>, index=<PyQt5.QtCore.QModelIndex object>)
92 def fetchMore(self, index: QModelIndex) -> None:
93 item = self.itemFromIndex(index)
---> 94 for child in item.yieldChildren():
item = <napari_omero.widgets.tree_model.OMEROTreeItem object at 0x1759263b0>
95 child_item = OMEROTreeItem(child)
96 item.appendRow(child_item)
File ~/Documents/dev/napari-omero/src/napari_omero/widgets/tree_model.py:24, in OMEROTreeItem.yieldChildren(self=<napari_omero.widgets.tree_model.OMEROTreeItem object>)
23 def yieldChildren(self):
---> 24 yield from self.wrapper._conn.getObjects(
self.wrapper._conn = <omero.gateway._BlitzGateway object at 0x1759cae30>
self = <napari_omero.widgets.tree_model.OMEROTreeItem object at 0x1759263b0>
self.wrapper = <_ProjectWrapper id=1901>
25 self.child_type,
26 opts={
27 self.wrapper_type.lower(): self.wrapper.id,
28 "order_by": "obj.name",
29 },
30 )
File ~/micromamba/envs/napari-omero/lib/python3.10/site-packages/omero/gateway/__init__.py:3309, in _BlitzGateway.getObjects(self=<omero.gateway._BlitzGateway object>, obj_type='Dataset', ids=None, params=object #0 (::omero::sys::Parameters)
{
map =... }
theFilter = <nil>
theOptions = <nil>
}, attributes=None, respect_order=False, opts={'order_by': 'obj.name', 'project': 1901})
3306 query, params, wrapper = self.buildQuery(
3307 obj_type, ids, params, attributes, opts)
3308 qs = self.getQueryService()
-> 3309 result = qs.findAllByQuery(query, params, self.SERVICE_OPTS)
qs = <omero.gateway.ProxyObjectWrapper object at 0x1759cb3a0>
query = 'select obj from Dataset obj join fetch obj.details.owner as owner join fetch obj.details.creationEvent join obj.projectLinks projectLinks where projectLinks.parent.id = :pid order by obj.name, obj.id'
params = object #0 (::omero::sys::Parameters)
{
map =
{
key = pid
value = object #1 (::omero::RLong)
{
_val = 1901
}
}
theFilter = <nil>
theOptions = <nil>
}
self.SERVICE_OPTS = <ServiceOptsDict: {'omero.client.uuid': '26e367ce-f620-413c-8cf2-d8d26b4eb5e4', 'omero.session.uuid': 'a26417f0-1cf1-478f-9f06-7c1b5a8ce158', 'omero.group': '-1'}>
self = <omero.gateway._BlitzGateway object at 0x1759cae30>
3310 if respect_order and ids is not None:
3311 idMap = {}
File ~/micromamba/envs/napari-omero/lib/python3.10/site-packages/omero/gateway/__init__.py:4862, in OmeroGatewaySafeCallWrapper.__call__(self=<omero.gateway.OmeroGatewaySafeCallWrapper object>, *args=('select obj from Dataset obj join fetch obj.detai...tLinks.parent.id = :pid order by obj.name, obj.id', object #0 (::omero::sys::Parameters)
{
map =... }
theFilter = <nil>
theOptions = <nil>
}, <ServiceOptsDict: {'omero.client.uuid': '26e367c...f1-478f-9f06-7c1b5a8ce158', 'omero.group': '-1'}>), **kwargs={})
4860 except Exception as e:
4861 self.debug(e.__class__.__name__, args, kwargs)
-> 4862 return self.handle_exception(e, *args, **kwargs)
self = <omero.gateway.OmeroGatewaySafeCallWrapper object at 0x17fc219c0>
args = ('select obj from Dataset obj join fetch obj.details.owner as owner join fetch obj.details.creationEvent join obj.projectLinks projectLinks where projectLinks.parent.id = :pid order by obj.name, obj.id', object #0 (::omero::sys::Parameters)
{
map =
{
key = pid
value = object #1 (::omero::RLong)
{
_val = 1901
}
}
theFilter = <nil>
theOptions = <nil>
}, <ServiceOptsDict: {'omero.client.uuid': '26e367ce-f620-413c-8cf2-d8d26b4eb5e4', 'omero.session.uuid': 'a26417f0-1cf1-478f-9f06-7c1b5a8ce158', 'omero.group': '-1'}>)
kwargs = {}
File ~/micromamba/envs/napari-omero/lib/python3.10/site-packages/omero/gateway/__init__.py:4859, in OmeroGatewaySafeCallWrapper.__call__(self=<omero.gateway.OmeroGatewaySafeCallWrapper object>, *args=('select obj from Dataset obj join fetch obj.detai...tLinks.parent.id = :pid order by obj.name, obj.id', object #0 (::omero::sys::Parameters)
{
map =... }
theFilter = <nil>
theOptions = <nil>
}, <ServiceOptsDict: {'omero.client.uuid': '26e367c...f1-478f-9f06-7c1b5a8ce158', 'omero.group': '-1'}>), **kwargs={})
4857 def __call__(self, *args, **kwargs):
4858 try:
-> 4859 return self.f(*args, **kwargs)
self.f = <bound method IQueryPrx.findAllByQuery of a26417f0-1cf1-478f-9f06-7c1b5a8ce158/26e367ce-f620-413c-8cf2-d8d26b4eb5e4omero.api.IQuery -t -e 1.1:tcp -h 192.168.228.191 -p 35511 -t 60000>
self = <omero.gateway.OmeroGatewaySafeCallWrapper object at 0x17fc219c0>
args = ('select obj from Dataset obj join fetch obj.details.owner as owner join fetch obj.details.creationEvent join obj.projectLinks projectLinks where projectLinks.parent.id = :pid order by obj.name, obj.id', object #0 (::omero::sys::Parameters)
{
map =
{
key = pid
value = object #1 (::omero::RLong)
{
_val = 1901
}
}
theFilter = <nil>
theOptions = <nil>
}, <ServiceOptsDict: {'omero.client.uuid': '26e367ce-f620-413c-8cf2-d8d26b4eb5e4', 'omero.session.uuid': 'a26417f0-1cf1-478f-9f06-7c1b5a8ce158', 'omero.group': '-1'}>)
kwargs = {}
4860 except Exception as e:
4861 self.debug(e.__class__.__name__, args, kwargs)
File ~/micromamba/envs/napari-omero/lib/python3.10/site-packages/omero_api_IQuery_ice.py:710, in IQueryPrx.findAllByQuery(self=a26417f0-1cf1-478f-9f06-7c1b5a8ce158/26e367ce-f6...t -e 1.1:tcp -h 192.168.228.191 -p 35511 -t 60000, query='select obj from Dataset obj join fetch obj.detai...tLinks.parent.id = :pid order by obj.name, obj.id', params=object #0 (::omero::sys::Parameters)
{
map =... }
theFilter = <nil>
theOptions = <nil>
}, _ctx=<ServiceOptsDict: {'omero.client.uuid': '26e367c...f1-478f-9f06-7c1b5a8ce158', 'omero.group': '-1'}>)
709 def findAllByQuery(self, query, params, _ctx=None):
--> 710 return _M_omero.api.IQuery._op_findAllByQuery.invoke(self, ((query, params), _ctx))
self = a26417f0-1cf1-478f-9f06-7c1b5a8ce158/26e367ce-f620-413c-8cf2-d8d26b4eb5e4omero.api.IQuery -t -e 1.1:tcp -h 192.168.228.191 -p 35511 -t 60000
((query, params), _ctx) = (('select obj from Dataset obj join fetch obj.details.owner as owner join fetch obj.details.creationEvent join obj.projectLinks projectLinks where projectLinks.parent.id = :pid order by obj.name, obj.id', object #0 (::omero::sys::Parameters)
{
map =
{
key = pid
value = object #1 (::omero::RLong)
{
_val = 1901
}
}
theFilter = <nil>
theOptions = <nil>
}), <ServiceOptsDict: {'omero.client.uuid': '26e367ce-f620-413c-8cf2-d8d26b4eb5e4', 'omero.session.uuid': 'a26417f0-1cf1-478f-9f06-7c1b5a8ce158', 'omero.group': '-1'}>)
_M_omero.api.IQuery._op_findAllByQuery = <IcePy.Operation object at 0x1032797b0>
(query, params) = ('select obj from Dataset obj join fetch obj.details.owner as owner join fetch obj.details.creationEvent join obj.projectLinks projectLinks where projectLinks.parent.id = :pid order by obj.name, obj.id', object #0 (::omero::sys::Parameters)
{
map =
{
key = pid
value = object #1 (::omero::RLong)
{
_val = 1901
}
}
theFilter = <nil>
theOptions = <nil>
})
_ctx = <ServiceOptsDict: {'omero.client.uuid': '26e367ce-f620-413c-8cf2-d8d26b4eb5e4', 'omero.session.uuid': 'a26417f0-1cf1-478f-9f06-7c1b5a8ce158', 'omero.group': '-1'}>
query = 'select obj from Dataset obj join fetch obj.details.owner as owner join fetch obj.details.creationEvent join obj.projectLinks projectLinks where projectLinks.parent.id = :pid order by obj.name, obj.id'
params = object #0 (::omero::sys::Parameters)
{
map =
{
key = pid
value = object #1 (::omero::RLong)
{
_val = 1901
}
}
theFilter = <nil>
theOptions = <nil>
}
_M_omero.api = <module 'omero.api' from '/Users/sobolp/micromamba/envs/napari-omero/lib/python3.10/site-packages/omero/api/__init__.py'>
_M_omero = <module 'omero' from '/Users/sobolp/micromamba/envs/napari-omero/lib/python3.10/site-packages/omero/__init__.py'>
ConnectionLostException: Ice.ConnectionLostException:
recv() returned zero
WARNING:omero.gateway:ConnectionLostException on <class 'omero.gateway.OmeroGatewaySafeCallWrapper'> to <26e367ce-f620-413c-8cf2-d8d26b4eb5e4omero.api.IQuery> findAllByQuery(('select obj from Dataset obj join fetch obj.details.owner as owner join fetch obj.details.creationEvent join obj.projectLinks projectLinks where projectLinks.parent.id = :pid order by obj.name, obj.id', object #0 (::omero::sys::Parameters)
{
map =
{
key = pid
value = object #1 (::omero::RLong)
{
_val = 1901
}
}
theFilter = <nil>
theOptions = <nil>
}, <ServiceOptsDict: {'omero.client.uuid': '26e367ce-f620-413c-8cf2-d8d26b4eb5e4', 'omero.session.uuid': 'a26417f0-1cf1-478f-9f06-7c1b5a8ce158', 'omero.group': '-1'}>), {})
Traceback (most recent call last):
File "/Users/sobolp/micromamba/envs/napari-omero/lib/python3.10/site-packages/omero/gateway/__init__.py", line 4859, in __call__
return self.f(*args, **kwargs)
File "/Users/sobolp/micromamba/envs/napari-omero/lib/python3.10/site-packages/omero_api_IQuery_ice.py", line 710, in findAllByQuery
return _M_omero.api.IQuery._op_findAllByQuery.invoke(self, ((query, params), _ctx))
Ice.ConnectionLostException: Ice.ConnectionLostException:
recv() returned zero
I ran into the same problem recently, you beat me to creating the issue. One way of fixing it would be to start a thread that checks whether the connection is still open at a regular time interval. If that would return False, the thread would have to trigger the teardown action associated to the Disconnect button and exit itself.
Unfortunately, I don't know how to edit the timeout of the omero Python client so my only way of testing this at the moment is to sit around for ~10minutes to see whether the reconnect faills ^^"
Edit: Maybe this fixes it.
@jo-mueller: happy to get you setup with a test server that has reduced connection timeout settings if it'll help.
https://omero.readthedocs.io/en/stable/sysadmins/config.html#omero-sessions-timeout e.g. is a place to start, e.g., by setting CONFIG_omero_sessions_timeout=60000 on the omero server docker: https://github.com/ome/omero-server-docker/blob/master/README.md#omeroserver-docker
Hi @joshmoore thanks for chiming in. I have also set myself up a dockerized omero test instance - good to know how to set the timeout on the serverside 👍
I think for testing purposes and to see how the plugin handles timeout, I'll set it to something really low and take it from there. Hope this helps to engineer a clean exit in case the gateway does timeout, anyway.
Sounds good. There's a secondary timeout within the glacier2router process, but I assume you're hitting the server-side one.
@joshmoore
happy to get you setup with a test server that has reduced connection timeout settings if it'll help.
On that note: Do you think there's any possibility to have such an omero instance running to be able to do some proper CI testing for omero-based tools? From what I've seen here but also at omero-py or ezomero is that it's terribly hard to maintain good coverage and testing - probably due to not being able to test most functionality in a CI pipeline 🤔
ezomero runs an omero server within its tests... https://github.com/TheJacksonLaboratory/ezomero/blob/main/.github/workflows/run_tests_pr.yml
see the docker compose in /tests
I think, I have a rough working version, but what I'm lacking - what's a propper way to check whether the connection is still up? E.g., if I'd use ezomero to spin up a conn object similar to how napari-omero does it:
conn = ezomero.connect(...)
annd then turn off my VPN which I use to tunnel into my institution network or go to airplane mode on my laptop, the property conn.isConnected() never goes to False 🤔 Am I missing something in how the BlitzGateway actually works? I would expect that isConnected does a check under the hood whether the remote server still replies but apparently I'm misstaken there...
Omero-py version is 5.19.5, for reference
Ok, I think I got it. The BlitzGateway is a static object, so actively tracking its own connection is probably difficult. I built in a threaded function that checks continuously whether the host can be reached and triggers the disconnection if it doesn't.