ibis-bigquery icon indicating copy to clipboard operation
ibis-bigquery copied to clipboard

[BigQuery] `test_array_concat` is failing

Open tswast opened this issue 5 years ago • 1 comments

Failing test

https://github.com/ibis-project/ibis/blob/a70d443c7931cb8bb47c52f97999589566e03cb2/ibis/tests/all/test_array.py#L11-L15

Test output

$ pytest ibis/tests/all/test_array.py::test_array_concat[BigQuery]
=============================== test session starts ===============================
platform darwin -- Python 3.7.8, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /Users/swast/src/ibis, inifile: setup.cfg
plugins: forked-1.2.0, mock-3.1.1, cov-2.10.0, xdist-1.34.0
collected 1 item                                                                  

ibis/tests/all/test_array.py F                                              [100%]

==================================== FAILURES =====================================
___________________________ test_array_concat[BigQuery] ___________________________

backend = <ibis.tests.backends.BigQuery object at 0x7fc3cbea21d0>
con = <ibis.bigquery.client.BigQueryClient object at 0x7fc3cbea2850>

    @pytest.mark.xfail_unsupported
    @pytest.mark.skip_missing_feature(
        ['supports_arrays', 'supports_arrays_outside_of_select']
    )
    # Issues #
    #@pytest.mark.xfail_backends([BigQuery])
    def test_array_concat(backend, con):
        left = ibis.literal([1, 2, 3])
        right = ibis.literal([2, 1])
        expr = left + right
        result = con.execute(expr)
>       assert result == [1, 2, 3, 2, 1]
E       ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

ibis/tests/all/test_array.py:18: ValueError
============================= short test summary info =============================
FAILED ibis/tests/all/test_array.py::test_array_concat[BigQuery] - ValueError: T...
================================ 1 failed in 1.33s ================================

Thoughts on fix

I suspect the query is returning correct results, but arrays are now coming back as numpy arrays. This is probably because google-cloud-bigquery is now using Arrow as an intermediate format before converting to a DataFrame.

Possible fixes:

  • If a list object is required, add some additional conversion logic to convert arrays to lists.
  • If a numpy object is okay, update the test to use a more general elementwise comparison.

tswast avatar Sep 10 '20 14:09 tswast

Found in ibis-project/ibis#2353

tswast avatar Sep 10 '20 14:09 tswast