BUG: Vizier query for eclipsing binary tables gives 'unclosed token'

[keflavich]

This is already covered in our test suite.

from astroquery.vizier import Vizier
V = Vizier()
result = V.find_catalogs('eclipsing binary', max_catalogs=5000)

results in

WARNING: UnitsWarning: Unit 'JD' not supported by the VOUnit standard.  [astropy.units.format.vounit]
WARNING: UnitsWarning: Unit 'Sun' not supported by the VOUnit standard. Did you mean uN? [astropy.units.format.vounit]
WARNING: UnitsWarning: The unit 'pix' has been deprecated in the VOUnit standard. [astropy.units.format.utils]
WARNING: UnitsWarning: Unit 'Rgeo' not supported by the VOUnit standard.  [astropy.units.format.vounit]
WARNING: UnitsWarning: Unit 'ppm' not supported by the VOUnit standard. Did you mean Pm or pm? [astropy.units.format.vounit]
WARNING: UnitsWarning: Unit 'Earth' not supported by the VOUnit standard.  [astropy.units.format.vounit]
WARNING: UnitsWarning: The unit 'ct' has been deprecated in the VOUnit standard. [astropy.units.format.utils]
WARNING: UnitsWarning: Unit 'Mgeo' not supported by the VOUnit standard.  [astropy.units.format.vounit]
WARNING: UnitsWarning: Unit 'Mjup' not supported by the VOUnit standard.  [astropy.units.format.vounit]
WARNING: UnitsWarning: Unit 'Rjup' not supported by the VOUnit standard.  [astropy.units.format.vounit]
WARNING: UnitsWarning: Unit 'month' not supported by the VOUnit standard.  [astropy.units.format.vounit]
WARNING: UnitsWarning: The unit 'au' has been deprecated in the VOUnit standard. Suggested: 1.4959787e+11m. [astropy.units.format.utils]
/Users/adam/repos/astroquery/astroquery/vizier/core.py:769: UserWarning: VOTABLE parsing raised exception: 106669:4: unclosed token
  warnings.warn("VOTABLE parsing raised exception: {0}".format(ex))
Traceback (most recent call last):
  File "/Users/adam/repos/astroquery/astroquery/vizier/core.py", line 767, in parse_vizier_votable
    vo_tree = votable.parse(tf, verify='warn', invalid='exception')
  File "/Users/adam/miniconda3/envs/py39forge/lib/python3.9/site-packages/astropy/utils/decorators.py", line 546, in wrapper
    return function(*args, **kwargs)
  File "/Users/adam/miniconda3/envs/py39forge/lib/python3.9/site-packages/astropy/io/votable/table.py", line 159, in parse
    return tree.VOTableFile(
  File "/Users/adam/miniconda3/envs/py39forge/lib/python3.9/site-packages/astropy/io/votable/tree.py", line 3615, in parse
    tag_mapping.get(tag, self._add_unknown_tag)(
  File "/Users/adam/miniconda3/envs/py39forge/lib/python3.9/site-packages/astropy/io/votable/tree.py", line 3494, in _add_resource
    resource.parse(self, iterator, config)
  File "/Users/adam/miniconda3/envs/py39forge/lib/python3.9/site-packages/astropy/io/votable/tree.py", line 3292, in parse
    tag_mapping.get(tag, self._add_unknown_tag)(
  File "/Users/adam/miniconda3/envs/py39forge/lib/python3.9/site-packages/astropy/io/votable/tree.py", line 3238, in _add_table
    table.parse(iterator, config)
  File "/Users/adam/miniconda3/envs/py39forge/lib/python3.9/site-packages/astropy/io/votable/tree.py", line 2459, in parse
    for start, tag, data, pos in iterator:
ValueError: 106669:4: unclosed token

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Users/adam/repos/astroquery/astroquery/vizier/core.py", line 699, in _parse_result
    return parse_vizier_votable(
  File "/Users/adam/repos/astroquery/astroquery/vizier/core.py", line 770, in parse_vizier_votable
    vo_tree = votable.parse(tf, verify='warn', invalid='mask')
  File "/Users/adam/miniconda3/envs/py39forge/lib/python3.9/site-packages/astropy/utils/decorators.py", line 546, in wrapper
    return function(*args, **kwargs)
  File "/Users/adam/miniconda3/envs/py39forge/lib/python3.9/site-packages/astropy/io/votable/table.py", line 159, in parse
    return tree.VOTableFile(
  File "/Users/adam/miniconda3/envs/py39forge/lib/python3.9/site-packages/astropy/io/votable/tree.py", line 3615, in parse
    tag_mapping.get(tag, self._add_unknown_tag)(
  File "/Users/adam/miniconda3/envs/py39forge/lib/python3.9/site-packages/astropy/io/votable/tree.py", line 3494, in _add_resource
    resource.parse(self, iterator, config)
  File "/Users/adam/miniconda3/envs/py39forge/lib/python3.9/site-packages/astropy/io/votable/tree.py", line 3292, in parse
    tag_mapping.get(tag, self._add_unknown_tag)(
  File "/Users/adam/miniconda3/envs/py39forge/lib/python3.9/site-packages/astropy/io/votable/tree.py", line 3238, in _add_table
    table.parse(iterator, config)
  File "/Users/adam/miniconda3/envs/py39forge/lib/python3.9/site-packages/astropy/io/votable/tree.py", line 2459, in parse
    for start, tag, data, pos in iterator:
ValueError: 106669:4: unclosed token

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<ipython-input-5-2e0f7bf34a93>", line 1, in <cell line: 1>
    result = V.find_catalogs('eclipsing binary', max_catalogs=5000)
  File "/Users/adam/repos/astroquery/astroquery/vizier/core.py", line 235, in find_catalogs
    result = self._parse_result(response, verbose=verbose,
  File "/Users/adam/repos/astroquery/astroquery/vizier/core.py", line 705, in _parse_result
    raise TableParseError("Failed to parse VIZIER result! The "
TableParseError: Failed to parse VIZIER result! The raw response can be found in self.response, and the error in self.table_parse_error. The attempted parsed result is in self.parsed_result.
 Exception: 106669:4: unclosed token

I think this is an unexpected failure. I'm investigating.

[keflavich]

This appears to be a truncated data stream:

In [11]: V.response.text[-1000:]
Out[11]: '" type="meta" nrows="1024">\n    <INFO name="nrows" value="1024">Number of rows of the table</INFO>\n    <DESCRIPTION>\n      BVRI observations of LU Lac\n    </DESCRIPTION>\n\n<!-- BVRI observations of LU Lac \\vizContent{timeSerie}(\\wGraph{J/other/NewA/31.65/./table1} {}{light curves}) -->\n    <LINK content-role="query" action="votable?-source=J/other/NewA/31.65/table1&amp;">\n      Action to Query the current RESOURCE, to be completed by PARAMs and FIELDs contents\n    </LINK>\n    <!-- Definitions of GROUPs and FIELDs -->\n\n<!-- +++No column could be found to attach a LINK in table: J/other/NewA/31.65/table1 -->\n    <FIELD name="recno" ucd="meta.record" display="0" datatype="int" width="8"><!-- ucd="RECORD" -->\n      <DESCRIPTION>Record number assigned by the VizieR team. Should Not be used for identification.</DESCRIPTION>\n      <LINK href="http://vizier.u-strasbg.fr/viz-bin/VizieR-5?-info=XML&amp;-out.add=.&amp;-source=J/other/NewA/31.65/table1&amp;recno=${recno}"/>\n    </FIELD>\n    <FIELD\n'

[keflavich]

I'll contact Vizier about it.

[keflavich]

Ah, maybe this is just a fundamental limit. We can replace the test with:

result = V.find_catalogs('eclipsing binary', max_catalogs=500)
Traceback (most recent call last):
  File "<ipython-input-12-593b0444fc4c>", line 1, in <cell line: 1>
    result = V.find_catalogs('eclipsing binary', max_catalogs=500)
  File "/Users/adam/repos/astroquery/astroquery/vizier/core.py", line 232, in find_catalogs
    raise ValueError("Maximum number of catalogs exceeded.  Try "
ValueError: Maximum number of catalogs exceeded.  Try setting max_catalogs to a large number and try again

that's the desired error.

[keflavich]

Nope, nope - max_catalogs is our internal limit. The original error is coming from a truncated very large data stream.

[gilleslandais]

Hello - a bug without any ambiguities..

I made a patch which fixes the truncated VOTable. However, there is still an error reported in the astroquery lines code. .... /usr/local/lib/python3.8/dist-packages/astroquery/vizier/core.py:723: UserWarning: VOTABLE parsing raised exception: 70401:78: unclosed token ....

the VOTAble URL is: (are there something expected or to fix ?) https://vizier.cds.unistra.fr/viz-bin/votable?-words=eclipsing binary&-meta.all=1&-ucd=&-meta.max=5000

• regards

[bsipocz]

All examples from here work now, thanks for all the upstream fixes!