Nightly tests are failing on Linux and Mac for Python 3.5

[pieths]

On July 19th, the Linux and Mac nightly test runs for Python 3.5 started failing with multiple errors which all look similar to:

_ ERROR collecting dependencies/Python3.5/lib/python3.5/site-packages/nimbusml/tests/feature_extraction/categorical/test_onehotvectorizer.py _
dependencies/Python3.5/lib/python3.5/site-packages/py/_path/local.py:701: in pyimport
    __import__(modname)
dependencies/Python3.5/lib/python3.5/site-packages/nimbusml/__init__.py:21: in <module>
    from .internal.utils.data_schema import DataSchema
dependencies/Python3.5/lib/python3.5/site-packages/nimbusml/internal/utils/data_schema.py:18: in <module>
    from pandas import DataFrame, Series, read_csv
dependencies/Python3.5/lib/python3.5/site-packages/pandas/__init__.py:55: in <module>
    from pandas.core.api import (
dependencies/Python3.5/lib/python3.5/site-packages/pandas/core/api.py:5: in <module>
    from pandas.core.arrays.integer import (
dependencies/Python3.5/lib/python3.5/site-packages/pandas/core/arrays/__init__.py:1: in <module>
    from .array_ import array  # noqa: F401
dependencies/Python3.5/lib/python3.5/site-packages/pandas/core/arrays/array_.py:7: in <module>
    from pandas.core.dtypes.common import (
dependencies/Python3.5/lib/python3.5/site-packages/pandas/core/dtypes/common.py:11: in <module>
    from pandas.core.dtypes.dtypes import (
dependencies/Python3.5/lib/python3.5/site-packages/pandas/core/dtypes/dtypes.py:53: in <module>
    class Registry:
dependencies/Python3.5/lib/python3.5/site-packages/pandas/core/dtypes/dtypes.py:84: in Registry
    self, dtype: Union[Type[ExtensionDtype], str]
dependencies/Python3.5/lib/python3.5/typing.py:552: in __getitem__
    dict(self.__dict__), parameters, _root=True)
dependencies/Python3.5/lib/python3.5/typing.py:512: in __new__
    for t2 in all_params - {t1} if not isinstance(t2, TypeVar)):
dependencies/Python3.5/lib/python3.5/typing.py:512: in <genexpr>
    for t2 in all_params - {t1} if not isinstance(t2, TypeVar)):
dependencies/Python3.5/lib/python3.5/typing.py:1077: in __subclasscheck__
    if super().__subclasscheck__(cls):
dependencies/Python3.5/lib/python3.5/abc.py:225: in __subclasscheck__
    for scls in cls.__subclasses__():
E   TypeError: descriptor '__subclasses__' of 'type' object needs an argument

[pieths]

I was able to repro the issue using Ubuntu 18.04 in WSL (Windows Subsystem for Linux) v1.

It looks like this is a known issue with the Python 3.5 series. See here. According to that issue, the bug might have been fixed in Python >= 3.5.3. Though there is a comment that claims to still see the issue with Python 3.5.6. The NimbusML build system on Linux is currently using Python 3.5.2. This information can be retrieved with:

$ ./dependencies/Python3.5/bin/python --version
Python 3.5.2 :: Anaconda 4.2.0 (64-bit)

It looks like the error might be related to the pandas package. The pandas version in the nightly builds got upgraded on July 19th. It went from 0.24.2 to 0.25.0.

I temporarily updated the build.sh script to force the use of pandas 0.24.2 with,

    ...
    "${PythonExe}" -m pip install --upgrade "${Wheel}"
    "${PythonExe}" -m pip install "scikit-learn==0.19.2"
    "${PythonExe}" -m pip install "pandas==0.24.2"    # <====== ADDED

    PackagePath=${PythonRoot}/lib/python${PythonVersion}/site-packages/nimbusml
    TestsPath1=${PackagePath}/tests
    ...

and this got rid of the error.

According to the release notes for pandas 0.25.x:

Starting with the 0.25.x series of releases, pandas only supports Python 3.5.3 and higher.

It seems like it might be time to update the Python version that is being used for the Linux and Mac builds.

[pieths]

See the How do I get Anaconda with Python 3.5 or 3.6? section in the Anaconda FAQ for few ways this can be done.