napari-omero icon indicating copy to clipboard operation
napari-omero copied to clipboard

Access to large images (>2GB)

Open perlman opened this issue 4 years ago • 1 comments

The napari-omero plugin fails to load large images from OMERO.

This is not too surprising given the use of PixelService to read image data. This image size is 230144x96256x3.

I think this will be addressed with eventual pyramid support as mentioned in #1 .

napari Image:26434
14:18:46 ERROR Failed to getPlane() or getTile() from rawPixelsStore
Traceback (most recent call last):
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/omero/gateway/__init__.py", line 7466, in getTiles
    rawPlane = rawPixelsStore.getPlane(z, c, t)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/omero_api_RawPixelsStore_ice.py", line 1199, in getPlane
    return _M_omero.api.RawPixelsStore._op_getPlane.invoke(self, ((z, c, t), _ctx))
omero.ApiUsageException: exception ::omero::ApiUsageException
{
    serverStackTrace = ome.conditions.ApiUsageException: Converting Long 22152740864 to Integer is an overflow.
        at ome.io.nio.RomioPixelBuffer.safeLongToInteger(RomioPixelBuffer.java:123)
        at ome.services.RawPixelsBean.getPlane(RawPixelsBean.java:392)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.tools.hibernate.SessionHandler.doStateful(SessionHandler.java:216)
        at ome.tools.hibernate.SessionHandler.invoke(SessionHandler.java:200)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy113.getPlane(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy113.getPlane(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172)
        at ome.services.throttling.Callback.run(Callback.java:56)
        at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
        at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140)
        at ome.services.blitz.impl.RawPixelsStoreI.getPlane_async(RawPixelsStoreI.java:105)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at omero.cmd.CallContext.invoke(CallContext.java:85)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy114.getPlane_async(Unknown Source)
        at omero.api._RawPixelsStoreTie.getPlane_async(_RawPixelsStoreTie.java:150)
        at omero.api._RawPixelsStoreDisp.___getPlane(_RawPixelsStoreDisp.java:1107)
        at omero.api._RawPixelsStoreDisp.__dispatch(_RawPixelsStoreDisp.java:1639)
        at IceInternal.Incoming.invoke(Incoming.java:221)
        at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
        at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
        at Ice.ConnectionI.message(ConnectionI.java:1056)
        at IceInternal.ThreadPool.run(ThreadPool.java:395)
        at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
        at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
        at java.base/java.lang.Thread.run(Thread.java:834)

    serverExceptionClass = ome.conditions.ApiUsageException
    message = Converting Long 22152740864 to Integer is an overflow.
}
14:18:46 ERROR Unhandled exception:
Traceback (most recent call last):
  File "/Users/eric/anaconda3/envs/jax/bin/napari", line 8, in <module>
    sys.exit(main())
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/__main__.py", line 181, in main
    view_path(
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/view_layers.py", line 9, in view_path
    (where <layer_type> is replaced with one of the layer types):
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/components/add_layers_mixin.py", line 847, in open
    self._add_layers_with_plugins(
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/components/add_layers_mixin.py", line 914, in _add_layers_with_plugins
    new = self._add_layer_from_data(*_data)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/components/add_layers_mixin.py", line 986, in _add_layer_from_data
    layer = add_method(data, **(meta or {}))
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/components/add_layers_mixin.py", line 232, in add_image
    layer = self.add_layer(layers.Image(image, **i_kwargs))
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/layers/image/image.py", line 247, in __init__
    self._update_dims()
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/layers/base/base.py", line 406, in _update_dims
    self.refresh()
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/layers/base/base.py", line 655, in refresh
    self.set_view_slice()
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/layers/base/base.py", line 608, in set_view_slice
    self._set_view_slice()
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/layers/image/image.py", line 554, in _set_view_slice
    image = np.asarray(self.data[self.dims.indices]).transpose(order)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/numpy/core/_asarray.py", line 83, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/array/core.py", line 1364, in __array__
    x = self.compute()
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/base.py", line 167, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/base.py", line 452, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/threaded.py", line 76, in get
    results = get_async(
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/local.py", line 486, in get_async
    raise_exception(exc, tb)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/local.py", line 316, in reraise
    raise exc
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari_omero/utils.py", line 39, in wrapper_timer
    value = func(*args, **kwargs)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari_omero/plugins/loaders.py", line 119, in <lambda>
    get_plane = delayed(timer(lambda idx: pixels.getPlane(*idx)))
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/omero/gateway/__init__.py", line 7432, in getPlane
    planeList = list(self.getPlanes([(theZ, theC, theT)]))
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/omero/gateway/__init__.py", line 7499, in getTiles
    raise exc
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/omero/gateway/__init__.py", line 7466, in getTiles
    rawPlane = rawPixelsStore.getPlane(z, c, t)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/omero_api_RawPixelsStore_ice.py", line 1199, in getPlane
    return _M_omero.api.RawPixelsStore._op_getPlane.invoke(self, ((z, c, t), _ctx))
omero.ApiUsageException: exception ::omero::ApiUsageException
{
    serverStackTrace = ome.conditions.ApiUsageException: Converting Long 22152740864 to Integer is an overflow.
        at ome.io.nio.RomioPixelBuffer.safeLongToInteger(RomioPixelBuffer.java:123)
        at ome.services.RawPixelsBean.getPlane(RawPixelsBean.java:392)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.tools.hibernate.SessionHandler.doStateful(SessionHandler.java:216)
        at ome.tools.hibernate.SessionHandler.invoke(SessionHandler.java:200)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy113.getPlane(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy113.getPlane(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172)
        at ome.services.throttling.Callback.run(Callback.java:56)
        at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
        at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140)
        at ome.services.blitz.impl.RawPixelsStoreI.getPlane_async(RawPixelsStoreI.java:105)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at omero.cmd.CallContext.invoke(CallContext.java:85)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy114.getPlane_async(Unknown Source)
        at omero.api._RawPixelsStoreTie.getPlane_async(_RawPixelsStoreTie.java:150)
        at omero.api._RawPixelsStoreDisp.___getPlane(_RawPixelsStoreDisp.java:1107)
        at omero.api._RawPixelsStoreDisp.__dispatch(_RawPixelsStoreDisp.java:1639)
        at IceInternal.Incoming.invoke(Incoming.java:221)
        at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
        at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
        at Ice.ConnectionI.message(ConnectionI.java:1056)
        at IceInternal.ThreadPool.run(ThreadPool.java:395)
        at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
        at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
        at java.base/java.lang.Thread.run(Thread.java:834)

    serverExceptionClass = ome.conditions.ApiUsageException
    message = Converting Long 22152740864 to Integer is an overflow.
}

perlman avatar Sep 28 '20 18:09 perlman

My assumption would also be that the strategy from #1 would solve this.

joshmoore avatar Sep 29 '20 07:09 joshmoore