NASA Exoplanet Archive: test_api_tables fails on 32-bit architectures

[ViviCoder]

The following error occurs when running tests on a 32-bit Debian system:

 ============================= test session starts ==============================
platform linux -- Python 3.10.4, pytest-6.2.5, py-1.10.0, pluggy-1.0.0

Running tests with astroquery version 0.4.6_testrun.

Running tests with astropy_helpers version 4.0.1.
Running tests in /usr/lib/python3/dist-packages/astroquery.

Date: 2022-06-09T06:33:37

Platform: Linux-5.10.0-14-cloud-amd64-i686-with-glibc2.33

Executable: /usr/bin/python3

Full Python Version: 
3.10.4 (main, Mar 24 2022, 13:07:27) [GCC 11.2.0]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Package versions: 
Numpy: 1.21.5
Matplotlib: 3.5.1
Astropy: 5.0.4
APLpy: not available
pyregion: not available
regions: not available
pyVO: 1.2.1
mocpy: not available
astropy-healpix: not available
vamdclib: not available

=================================== FAILURES ===================================
_________________________ test_api_tables[toi-query28] _________________________

patch_get = <_pytest.monkeypatch.MonkeyPatch object at 0xed9ada48>
table = 'toi', query = {'where': 'toi=256.01'}

    @pytest.mark.parametrize("table,query", API_TABLES)
    def test_api_tables(patch_get, table, query):
        NasaExoplanetArchiveMock = NasaExoplanetArchiveClass()
    
        NasaExoplanetArchiveMock._tap_tables = ['list']
>       data = NasaExoplanetArchiveMock.query_criteria(table, select="*", **query)

/usr/lib/python3/dist-packages/astroquery/ipac/nexsci/nasa_exoplanet_archive/tests/test_nasa_exoplanet_archive.py:216: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3/dist-packages/astroquery/utils/class_or_instance.py:25: in f
    return self.fn(obj, *args, **kwds)
/usr/lib/python3/dist-packages/astroquery/utils/process_asyncs.py:29: in newmethod
    result = self._parse_result(response, verbose=verbose)
/usr/lib/python3/dist-packages/astroquery/ipac/nexsci/nasa_exoplanet_archive/core.py:609: in _parse_result
    data = ascii.read(text, format="ipac", fast_reader=False, converters=CONVERTERS)
/usr/lib/python3/dist-packages/astropy/io/ascii/ui.py:376: in read
    dat = reader.read(table)
/usr/lib/python3/dist-packages/astropy/io/ascii/core.py:1381: in read
    table = self.outputter(self.header.cols, self.meta)
/usr/lib/python3/dist-packages/astropy/io/ascii/core.py:1117: in __call__
    self._convert_vals(cols)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <astropy.io.ascii.core.TableOutputter object at 0xeda71a30>
cols = [<astropy.io.ascii.core.Column object at 0xeda71b08>, <astropy.io.ascii.core.Column object at 0xeda719a0>, <astropy.io...a718c8>, <astropy.io.ascii.core.Column object at 0xeda71c28>, <astropy.io.ascii.core.Column object at 0xeda71b98>, ...]

    def _convert_vals(self, cols):
        for col in cols:
            for key, converters in self.converters.items():
                if fnmatch.fnmatch(col.name, key):
                    break
            else:
                if col.dtype is not None:
                    converters = [convert_numpy(col.dtype)]
                else:
                    converters = self.default_converters
    
            col.converters = self._validate_and_copy(col, converters)
    
            # Catch the last error in order to provide additional information
            # in case all attempts at column conversion fail.  The initial
            # value of of last_error will apply if no converters are defined
            # and the first col.converters[0] access raises IndexError.
            last_err = 'no converters defined'
    
            while not hasattr(col, 'data'):
                # Try converters, popping the unsuccessful ones from the list.
                # If there are no converters left here then fail.
                if not col.converters:
                    raise ValueError(f'Column {col.name} failed to convert: {last_err}')
    
                converter_func, converter_type = col.converters[0]
                if not issubclass(converter_type, col.type):
>                   raise TypeError('converter type does not match column type')
E                   TypeError: converter type does not match column type

/usr/lib/python3/dist-packages/astropy/io/ascii/core.py:1064: TypeError

[bsipocz]

Thanks @ViviCoder for the report. Do you have any tips how to reproduce this locally?

cc @rickynilsson

[ViviCoder]

@bsipocz If you do not have a 32-bit machine, you can install a 32-bit Linux distribution (architecture i386 in Debian, for instance) on a 64-bit machine. This machine, of course, can be virtual.

[olebole]

I am trying to dig into this. Sometimes I explicitly see an OverflowError here:

self = <astropy.io.ascii.core.TableOutputter object at 0xed504760>
cols = [<astropy.io.ascii.core.Column object at 0xed504940>, <astropy.io.ascii.core.Column object at 0xed5045b0>, <astropy.i
o...504f40>, <astropy.io.ascii.core.Column object at 0xed504850>, <astropy.io.ascii.core.Column object at 0xed504dd8>, ...]

    def _convert_vals(self, cols):
        for col in cols:
            for key, converters in self.converters.items():
                if fnmatch.fnmatch(col.name, key):
                    break
            else:
                if col.dtype is not None:
                    converters = [convert_numpy(col.dtype)]
                else:
                    converters = self.default_converters
    
            col.converters = self._validate_and_copy(col, converters)
    
            # …
            last_err = 'no converters defined'
    
            while not hasattr(col, 'data'):
                # Try converters, popping the unsuccessful ones from the list.
                # If there are no converters left here then fail.
                if not col.converters:
                    raise ValueError(f'Column {col.name} failed to convert: {last_err}')
    
                converter_func, converter_type = col.converters[0]
                if not issubclass(converter_type, col.type):
                    raise TypeError('converter type does not match column type')
    
                try:
>                   col.data = converter_func(col.str_vals)

/usr/lib/python3/dist-packages/astropy/io/ascii/core.py:1089: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

vals = ['9048843364125']

    def generic_converter(vals):
>       return numpy.array(vals, numpy_type)
E       OverflowError: Python int too large to convert to C long

/usr/lib/python3/dist-packages/astropy/io/ascii/core.py:1008: OverflowError

So probably the cause is this part of astropy/io/ascii/core.py (lines 1096ff):

               except OverflowError as err:
                    # Overflow during conversion (most likely an int that                                                   
                    # doesn't fit in native C long). Put string at the top of                                               
                    # the converters list for the next while iteration.                                                     
                    warnings.warn(
                        "OverflowError converting to {} in column {}, reverting to String."
                        .format(converter_type.__name__, col.name), AstropyWarning)
                    col.converters.insert(0, convert_numpy(numpy.str))
                    last_err = err

This inserts a string converter as last hope, which then fails due to the incompatibility with an int column. The numpy_type in _convert_vals() is <class 'int'>. I would somehow guess that the column is just too big, and this cannot be run with 32 bit? But, wouldn't it be better if the numpy_type would explicitly be 64 bit wide? That comes from here, in astropy/io/ascii/core.py:1136:

class TableOutputter(BaseOutputter):
    """                                                                                                                     
    Output the table as an astropy.table.Table object.                                                                      
    """

    default_converters = [convert_numpy(int),  # <-***************
                          convert_numpy(float),
                          convert_numpy(str)]

Pragmatically replacing this with convert_numpy(numpy.int64) resolves this problem, so I guess it is an astropy bug?

[bsipocz]

Thanks @olebole for investigating, and fixing this upstream. (I've tried to find a 32bit server, but it's less and less easy, and VMing it on a M1 laptop didn't seem to be a reasonable idea).