omero-py
omero-py copied to clipboard
Tables 3.3+ locking problems
See investigation on https://github.com/ome/omero-py/pull/59#issuecomment-547913768
The 5.5.1 tables unit tests depend on the versions of tables and numpy. On CentOS7 the distribution versions are the only combination that passes:
- tables 3.2.0
- numpy 1.7.1
Results of testing the openmicroscopy 5.5.1 unit tests:
Dockerfile
:
FROM openmicroscopy/omero-server:5.5.1
USER root
RUN yum install -y -q \
future \
git \
hdf5-devel \
ice-all-devel \
java-1.8.0-openjdk-devel
USER omero-server
WORKDIR /opt/omero/server
RUN git clone --depth=1 https://github.com/ome/openmicroscopy -b v5.5.1
RUN virtualenv --system-site-packages build && \
build/bin/pip install -U pip setuptools
RUN . build/bin/activate && \
cd openmicroscopy && \
./build.py
$ . build/bin/activate
$ cd openmicroscopy/components/tools/OmeroPy
Default distribution packages:
$ ./setup.py test -vst test/unit/tablestest/
========================== 31 passed in 11.29 seconds ==========================
Tables 3.2.0
$ pip install tables==3.2
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting tables==3.2
Downloading https://files.pythonhosted.org/packages/92/3a/a0fd93a68b98ad2ab788f4f725250515ad2f12c23858f3b8d2b17af454a4/tables-3.2.0.tar.gz (7.0MB)
|################################| 7.0MB 2.7MB/s
Requirement already satisfied: numpy>=1.7.1 in /usr/lib64/python2.7/site-packages (from tables==3.2) (1.7.1)
Requirement already satisfied: numexpr>=2.4 in /usr/lib64/python2.7/site-packages (from tables==3.2) (2.5.2)
Building wheels for collected packages: tables
Building wheel for tables (setup.py) ... done
Created wheel for tables: filename=tables-3.2.0-cp27-cp27mu-linux_x86_64.whl size=3111313 sha256=3a985fa70fc64bae1c8fe653041d70ca0357d23a292e8efbc5569efd08438eaa
Stored in directory: /opt/omero/server/.cache/pip/wheels/65/6c/b5/eb899eac75a02682688572a6a47f14a00e19f4dc5604faf362
Successfully built tables
Installing collected packages: tables
Found existing installation: tables 3.2.0rc3
Not uninstalling tables at /usr/lib64/python2.7/site-packages, outside environment /opt/omero/server/build
Can't uninstall 'tables'. No files were found to uninstall.
Successfully installed tables-3.2.0
$ ./setup.py test -vst test/unit/tablestest/
========================== 31 passed in 6.49 seconds ===========================
Tables 3.3.0
$ pip install tables==3.3
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting tables==3.3
Downloading https://files.pythonhosted.org/packages/08/f5/a7d940a11c6fce0a0429f33e0de21e543a3c493b9afed6551f842fe6c97b/tables-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl (4.3MB)
|################################| 4.3MB 2.9MB/s
Requirement already satisfied: numexpr>=2.5.2 in /usr/lib64/python2.7/site-packages (from tables==3.3) (2.5.2)
Requirement already satisfied: six in /usr/lib/python2.7/site-packages (from tables==3.3) (1.9.0)
Collecting numpy>=1.8.0
Downloading https://files.pythonhosted.org/packages/d7/b1/3367ea1f372957f97a6752ec725b87886e12af1415216feec9067e31df70/numpy-1.16.5-cp27-cp27mu-manylinux1_x86_64.whl (17.0MB)
|################################| 17.0MB 22.1MB/s
Installing collected packages: numpy, tables
Found existing installation: numpy 1.7.1
Not uninstalling numpy at /usr/lib64/python2.7/site-packages, outside environment /opt/omero/server/build
Can't uninstall 'numpy'. No files were found to uninstall.
Found existing installation: tables 3.2.0
Uninstalling tables-3.2.0:
Successfully uninstalled tables-3.2.0
Successfully installed numpy-1.16.5 tables-3.3.0
$ ./setup.py test -vst test/unit/tablestest/
=================================== FAILURES ===================================
________________________ TestHdfStorage.testMaskColumn _________________________
self = <test_hdfstorage.TestHdfStorage object at 0x7f184ed61a50>
@pytest.mark.broken
def testMaskColumn(self):
hdf = HdfStorage(self.hdfpath(), self.lock)
mask = omero.columns.MaskColumnI('mask', 'desc', None)
hdf.initialize([mask], None)
mask.imageId = [1, 2]
mask.theZ = [2, 2]
mask.theT = [3, 3]
mask.x = [4, 4]
mask.y = [5, 5]
mask.w = [6, 6]
mask.h = [7, 7]
mask.bytes = [[0], [0, 1, 2, 3, 4]]
hdf.append([mask])
> data = hdf.readCoordinates(hdf._stamp, [0, 1], self.current)
test/unit/tablestest/test_hdfstorage.py:270:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
target/omero/util/decorators.py:90: in with_lock
return func(*args, **kwargs)
target/omero/hdfstorageV2.py:63: in check_and_update_stamp
return func(*args, **kwargs)
target/omero/hdfstorageV2.py:551: in readCoordinates
col.readCoordinates(self.__mea, rowNumbers)
target/omero/columns.py:464: in readCoordinates
self.getbytes(masks, rowNumbers)
target/omero/columns.py:476: in getbytes
self.bytes.append(masks[idx].tolist())
../../../../build/lib/python2.7/site-packages/tables/vlarray.py:671: in __getitem__
return self.read(start, stop, step)[0]
../../../../build/lib/python2.7/site-packages/tables/vlarray.py:811: in read
listarr = self._read_array(start, stop, step)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E ValueError: cannot set WRITEABLE flag to True of this array
tables/hdf5extension.pyx:2098: ValueError
===================== 1 failed, 30 passed in 7.01 seconds ======================
Tables 3.4.0
$ pip install tables==3.4
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting tables==3.4
Downloading https://files.pythonhosted.org/packages/90/31/db053cb5c82513cbcdadd2d6afaf885db3da36b3b705fe78443848fdec33/tables-3.4.0-cp27-cp27mu-manylinux1_x86_64.whl (4.4MB)
|################################| 4.4MB 2.9MB/s
Requirement already satisfied: numexpr>=2.5.2 in /usr/lib64/python2.7/site-packages (from tables==3.4) (2.5.2)
Requirement already satisfied: six in /usr/lib/python2.7/site-packages (from tables==3.4) (1.9.0)
Requirement already satisfied: numpy>=1.8.0 in /opt/omero/server/build/lib/python2.7/site-packages (from tables==3.4) (1.16.5)
Installing collected packages: tables
Found existing installation: tables 3.3.0
Uninstalling tables-3.3.0:
Successfully uninstalled tables-3.3.0
Successfully installed tables-3.4.0
$ ./setup.py test -vst test/unit/tablestest/
=================================== FAILURES ===================================
________________________ TestHdfStorage.testMaskColumn _________________________
self = <test_hdfstorage.TestHdfStorage object at 0x7f8455d33290>
@pytest.mark.broken
def testMaskColumn(self):
hdf = HdfStorage(self.hdfpath(), self.lock)
mask = omero.columns.MaskColumnI('mask', 'desc', None)
hdf.initialize([mask], None)
mask.imageId = [1, 2]
mask.theZ = [2, 2]
mask.theT = [3, 3]
mask.x = [4, 4]
mask.y = [5, 5]
mask.w = [6, 6]
mask.h = [7, 7]
mask.bytes = [[0], [0, 1, 2, 3, 4]]
hdf.append([mask])
> data = hdf.readCoordinates(hdf._stamp, [0, 1], self.current)
test/unit/tablestest/test_hdfstorage.py:270:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
target/omero/util/decorators.py:90: in with_lock
return func(*args, **kwargs)
target/omero/hdfstorageV2.py:63: in check_and_update_stamp
return func(*args, **kwargs)
target/omero/hdfstorageV2.py:551: in readCoordinates
col.readCoordinates(self.__mea, rowNumbers)
target/omero/columns.py:464: in readCoordinates
self.getbytes(masks, rowNumbers)
target/omero/columns.py:476: in getbytes
self.bytes.append(masks[idx].tolist())
../../../../build/lib/python2.7/site-packages/tables/vlarray.py:671: in __getitem__
return self.read(start, stop, step)[0]
../../../../build/lib/python2.7/site-packages/tables/vlarray.py:811: in read
listarr = self._read_array(start, stop, step)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E ValueError: cannot set WRITEABLE flag to True of this array
tables/hdf5extension.pyx:2106: ValueError
===================== 1 failed, 30 passed in 6.95 seconds ======================
Tables 3.5.2
$ pip install tables==3.5.2
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting tables==3.5.2
Downloading https://files.pythonhosted.org/packages/86/12/3e2dbdf79826a5e2e6a3bee363037db3a2a5f368435977f0999d75a7c6ee/tables-3.5.2-cp27-cp27mu-manylinux1_x86_64.whl (4.3MB)
|################################| 4.3MB 1.7MB/s
Collecting numexpr>=2.6.2
Downloading https://files.pythonhosted.org/packages/9e/e9/b72415d496b711baaf040098342b15ecbcceb0808325fc672f9c0177887b/numexpr-2.7.0-cp27-cp27mu-manylinux1_x86_64.whl (162kB)
|################################| 163kB 9.7MB/s
Collecting mock>=2.0
Downloading https://files.pythonhosted.org/packages/05/d2/f94e68be6b17f46d2c353564da56e6fb89ef09faeeff3313a046cb810ca9/mock-3.0.5-py2.py3-none-any.whl
Requirement already satisfied: numpy>=1.9.3 in /opt/omero/server/build/lib/python2.7/site-packages (from tables==3.5.2) (1.16.5)
Requirement already satisfied: six>=1.9.0 in /usr/lib/python2.7/site-packages (from tables==3.5.2) (1.9.0)
Collecting funcsigs>=1; python_version < "3.3"
Downloading https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl
Installing collected packages: numexpr, funcsigs, mock, tables
Found existing installation: numexpr 2.5.2
Not uninstalling numexpr at /usr/lib64/python2.7/site-packages, outside environment /opt/omero/server/build
Can't uninstall 'numexpr'. No files were found to uninstall.
Found existing installation: tables 3.4.0
Uninstalling tables-3.4.0:
Successfully uninstalled tables-3.4.0
Successfully installed funcsigs-1.0.2 mock-3.0.5 numexpr-2.7.0 tables-3.5.2
$ ./setup.py test -vst test/unit/tablestest/
=================================== FAILURES ===================================
___________________________ TestHdfList.testLocking ____________________________
self = <test_hdfstorage.TestHdfList object at 0x7fd93888da50>
monkeypatch = <_pytest.monkeypatch.monkeypatch instance at 0x7fd944905128>
def testLocking(self, monkeypatch):
lock1 = threading.RLock()
hdflist2 = HdfList()
lock2 = threading.RLock()
tmp = str(self.hdfpath())
# Using HDFLIST
hdf1 = HdfStorage(tmp, lock1)
# There are multiple guards against opening the same HDF5 file
# PyTables includes a check
monkeypatch.setattr(storage_module, 'HDFLIST', hdflist2)
with pytest.raises(ValueError) as exc_info:
> HdfStorage(tmp, lock2)
E Failed: DID NOT RAISE
test/unit/tablestest/test_hdfstorage.py:316: Failed
===================== 1 failed, 30 passed in 6.86 seconds ======================