omero-py Tables 3.3+ locking problems

Tables 3.3+ locking problems

Open manics opened this issue 4 years ago • 1 comments

See investigation on https://github.com/ome/omero-py/pull/59#issuecomment-547913768

The 5.5.1 tables unit tests depend on the versions of tables and numpy. On CentOS7 the distribution versions are the only combination that passes:

tables 3.2.0
numpy 1.7.1

Results of testing the openmicroscopy 5.5.1 unit tests: Dockerfile:

FROM openmicroscopy/omero-server:5.5.1

USER root
RUN yum install -y -q \
    future \
    git \
    hdf5-devel \
    ice-all-devel \
    java-1.8.0-openjdk-devel

USER omero-server
WORKDIR /opt/omero/server
RUN git clone --depth=1 https://github.com/ome/openmicroscopy -b v5.5.1
RUN virtualenv --system-site-packages build && \
    build/bin/pip install -U pip setuptools
RUN . build/bin/activate && \
    cd openmicroscopy && \
    ./build.py

$ . build/bin/activate
$ cd openmicroscopy/components/tools/OmeroPy

Default distribution packages:

$ ./setup.py test -vst test/unit/tablestest/

========================== 31 passed in 11.29 seconds ==========================

Tables 3.2.0

$ pip install tables==3.2
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting tables==3.2
  Downloading https://files.pythonhosted.org/packages/92/3a/a0fd93a68b98ad2ab788f4f725250515ad2f12c23858f3b8d2b17af454a4/tables-3.2.0.tar.gz (7.0MB)
     |################################| 7.0MB 2.7MB/s
Requirement already satisfied: numpy>=1.7.1 in /usr/lib64/python2.7/site-packages (from tables==3.2) (1.7.1)
Requirement already satisfied: numexpr>=2.4 in /usr/lib64/python2.7/site-packages (from tables==3.2) (2.5.2)
Building wheels for collected packages: tables
  Building wheel for tables (setup.py) ... done
  Created wheel for tables: filename=tables-3.2.0-cp27-cp27mu-linux_x86_64.whl size=3111313 sha256=3a985fa70fc64bae1c8fe653041d70ca0357d23a292e8efbc5569efd08438eaa
  Stored in directory: /opt/omero/server/.cache/pip/wheels/65/6c/b5/eb899eac75a02682688572a6a47f14a00e19f4dc5604faf362
Successfully built tables
Installing collected packages: tables
  Found existing installation: tables 3.2.0rc3
    Not uninstalling tables at /usr/lib64/python2.7/site-packages, outside environment /opt/omero/server/build
    Can't uninstall 'tables'. No files were found to uninstall.
Successfully installed tables-3.2.0

$ ./setup.py test -vst test/unit/tablestest/

========================== 31 passed in 6.49 seconds ===========================

Tables 3.3.0

$ pip install tables==3.3
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting tables==3.3
  Downloading https://files.pythonhosted.org/packages/08/f5/a7d940a11c6fce0a0429f33e0de21e543a3c493b9afed6551f842fe6c97b/tables-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl (4.3MB)
     |################################| 4.3MB 2.9MB/s
Requirement already satisfied: numexpr>=2.5.2 in /usr/lib64/python2.7/site-packages (from tables==3.3) (2.5.2)
Requirement already satisfied: six in /usr/lib/python2.7/site-packages (from tables==3.3) (1.9.0)
Collecting numpy>=1.8.0
  Downloading https://files.pythonhosted.org/packages/d7/b1/3367ea1f372957f97a6752ec725b87886e12af1415216feec9067e31df70/numpy-1.16.5-cp27-cp27mu-manylinux1_x86_64.whl (17.0MB)
     |################################| 17.0MB 22.1MB/s
Installing collected packages: numpy, tables
  Found existing installation: numpy 1.7.1
    Not uninstalling numpy at /usr/lib64/python2.7/site-packages, outside environment /opt/omero/server/build
    Can't uninstall 'numpy'. No files were found to uninstall.
  Found existing installation: tables 3.2.0
    Uninstalling tables-3.2.0:
      Successfully uninstalled tables-3.2.0
Successfully installed numpy-1.16.5 tables-3.3.0

$ ./setup.py test -vst test/unit/tablestest/

=================================== FAILURES ===================================
________________________ TestHdfStorage.testMaskColumn _________________________

self = <test_hdfstorage.TestHdfStorage object at 0x7f184ed61a50>

    @pytest.mark.broken
    def testMaskColumn(self):
        hdf = HdfStorage(self.hdfpath(), self.lock)
        mask = omero.columns.MaskColumnI('mask', 'desc', None)
        hdf.initialize([mask], None)
        mask.imageId = [1, 2]
        mask.theZ = [2, 2]
        mask.theT = [3, 3]
        mask.x = [4, 4]
        mask.y = [5, 5]
        mask.w = [6, 6]
        mask.h = [7, 7]
        mask.bytes = [[0], [0, 1, 2, 3, 4]]
        hdf.append([mask])
>       data = hdf.readCoordinates(hdf._stamp, [0, 1], self.current)

test/unit/tablestest/test_hdfstorage.py:270:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
target/omero/util/decorators.py:90: in with_lock
    return func(*args, **kwargs)
target/omero/hdfstorageV2.py:63: in check_and_update_stamp
    return func(*args, **kwargs)
target/omero/hdfstorageV2.py:551: in readCoordinates
    col.readCoordinates(self.__mea, rowNumbers)
target/omero/columns.py:464: in readCoordinates
    self.getbytes(masks, rowNumbers)
target/omero/columns.py:476: in getbytes
    self.bytes.append(masks[idx].tolist())
../../../../build/lib/python2.7/site-packages/tables/vlarray.py:671: in __getitem__
    return self.read(start, stop, step)[0]
../../../../build/lib/python2.7/site-packages/tables/vlarray.py:811: in read
    listarr = self._read_array(start, stop, step)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   ValueError: cannot set WRITEABLE flag to True of this array

tables/hdf5extension.pyx:2098: ValueError
===================== 1 failed, 30 passed in 7.01 seconds ======================

Tables 3.4.0

$ pip install tables==3.4
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting tables==3.4
  Downloading https://files.pythonhosted.org/packages/90/31/db053cb5c82513cbcdadd2d6afaf885db3da36b3b705fe78443848fdec33/tables-3.4.0-cp27-cp27mu-manylinux1_x86_64.whl (4.4MB)
     |################################| 4.4MB 2.9MB/s
Requirement already satisfied: numexpr>=2.5.2 in /usr/lib64/python2.7/site-packages (from tables==3.4) (2.5.2)
Requirement already satisfied: six in /usr/lib/python2.7/site-packages (from tables==3.4) (1.9.0)
Requirement already satisfied: numpy>=1.8.0 in /opt/omero/server/build/lib/python2.7/site-packages (from tables==3.4) (1.16.5)
Installing collected packages: tables
  Found existing installation: tables 3.3.0
    Uninstalling tables-3.3.0:
      Successfully uninstalled tables-3.3.0
Successfully installed tables-3.4.0

$ ./setup.py test -vst test/unit/tablestest/

=================================== FAILURES ===================================
________________________ TestHdfStorage.testMaskColumn _________________________

self = <test_hdfstorage.TestHdfStorage object at 0x7f8455d33290>

    @pytest.mark.broken
    def testMaskColumn(self):
        hdf = HdfStorage(self.hdfpath(), self.lock)
        mask = omero.columns.MaskColumnI('mask', 'desc', None)
        hdf.initialize([mask], None)
        mask.imageId = [1, 2]
        mask.theZ = [2, 2]
        mask.theT = [3, 3]
        mask.x = [4, 4]
        mask.y = [5, 5]
        mask.w = [6, 6]
        mask.h = [7, 7]
        mask.bytes = [[0], [0, 1, 2, 3, 4]]
        hdf.append([mask])
>       data = hdf.readCoordinates(hdf._stamp, [0, 1], self.current)

test/unit/tablestest/test_hdfstorage.py:270:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
target/omero/util/decorators.py:90: in with_lock
    return func(*args, **kwargs)
target/omero/hdfstorageV2.py:63: in check_and_update_stamp
    return func(*args, **kwargs)
target/omero/hdfstorageV2.py:551: in readCoordinates
    col.readCoordinates(self.__mea, rowNumbers)
target/omero/columns.py:464: in readCoordinates
    self.getbytes(masks, rowNumbers)
target/omero/columns.py:476: in getbytes
    self.bytes.append(masks[idx].tolist())
../../../../build/lib/python2.7/site-packages/tables/vlarray.py:671: in __getitem__
    return self.read(start, stop, step)[0]
../../../../build/lib/python2.7/site-packages/tables/vlarray.py:811: in read
    listarr = self._read_array(start, stop, step)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   ValueError: cannot set WRITEABLE flag to True of this array

tables/hdf5extension.pyx:2106: ValueError
===================== 1 failed, 30 passed in 6.95 seconds ======================

Tables 3.5.2

$ pip install tables==3.5.2
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting tables==3.5.2
  Downloading https://files.pythonhosted.org/packages/86/12/3e2dbdf79826a5e2e6a3bee363037db3a2a5f368435977f0999d75a7c6ee/tables-3.5.2-cp27-cp27mu-manylinux1_x86_64.whl (4.3MB)
     |################################| 4.3MB 1.7MB/s
Collecting numexpr>=2.6.2
  Downloading https://files.pythonhosted.org/packages/9e/e9/b72415d496b711baaf040098342b15ecbcceb0808325fc672f9c0177887b/numexpr-2.7.0-cp27-cp27mu-manylinux1_x86_64.whl (162kB)
     |################################| 163kB 9.7MB/s
Collecting mock>=2.0
  Downloading https://files.pythonhosted.org/packages/05/d2/f94e68be6b17f46d2c353564da56e6fb89ef09faeeff3313a046cb810ca9/mock-3.0.5-py2.py3-none-any.whl
Requirement already satisfied: numpy>=1.9.3 in /opt/omero/server/build/lib/python2.7/site-packages (from tables==3.5.2) (1.16.5)
Requirement already satisfied: six>=1.9.0 in /usr/lib/python2.7/site-packages (from tables==3.5.2) (1.9.0)
Collecting funcsigs>=1; python_version < "3.3"
  Downloading https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl
Installing collected packages: numexpr, funcsigs, mock, tables
  Found existing installation: numexpr 2.5.2
    Not uninstalling numexpr at /usr/lib64/python2.7/site-packages, outside environment /opt/omero/server/build
    Can't uninstall 'numexpr'. No files were found to uninstall.
  Found existing installation: tables 3.4.0
    Uninstalling tables-3.4.0:
      Successfully uninstalled tables-3.4.0
Successfully installed funcsigs-1.0.2 mock-3.0.5 numexpr-2.7.0 tables-3.5.2

$ ./setup.py test -vst test/unit/tablestest/

=================================== FAILURES ===================================
___________________________ TestHdfList.testLocking ____________________________

self = <test_hdfstorage.TestHdfList object at 0x7fd93888da50>
monkeypatch = <_pytest.monkeypatch.monkeypatch instance at 0x7fd944905128>

    def testLocking(self, monkeypatch):
        lock1 = threading.RLock()
        hdflist2 = HdfList()
        lock2 = threading.RLock()
        tmp = str(self.hdfpath())

        # Using HDFLIST
        hdf1 = HdfStorage(tmp, lock1)

        # There are multiple guards against opening the same HDF5 file

        # PyTables includes a check
        monkeypatch.setattr(storage_module, 'HDFLIST', hdflist2)
        with pytest.raises(ValueError) as exc_info:
>           HdfStorage(tmp, lock2)
E           Failed: DID NOT RAISE

test/unit/tablestest/test_hdfstorage.py:316: Failed
===================== 1 failed, 30 passed in 6.86 seconds ======================

Oct 30 '19 17:10 manics

omero-py omero-py copied to clipboard

Tables 3.3+ locking problems

Default distribution packages:

Tables 3.2.0

Tables 3.3.0

Tables 3.4.0

Tables 3.5.2

omero-py
omero-py copied to clipboard