pl.Concat of empty LazyFrames fails

[Unigurd]

Checks

• [X] I have checked that this issue has not already been reported.
• [X] I have confirmed this bug exists on the latest version of Polars.

Reproducible example

I have found a couple of examples. The third one only fails sometimes.

Case 1

import polars as pl
from polars.testing import assert_frame_equal

df0 = pl.LazyFrame()
df1 = pl.LazyFrame({"a": []})
df2 = pl.concat([df0, df1])

assert_frame_equal(df2, df2.collect().lazy())

Case 2

import polars as pl
from polars.testing import assert_frame_equal

file = "/tmp/somefile.parquet"

df0 = pl.LazyFrame()
df1 = pl.LazyFrame({"a": [1, 2, 3]})
df2 = pl.concat([df0, df1])

df2.sink_parquet(file)
df3 = pl.scan_parquet(file)

assert_frame_equal(df2, df3)

Case 3

import polars as pl
from polars.testing import assert_frame_equal

file = "/tmp/somefile.parquet"

df0 = pl.LazyFrame(schema={"a": pl.Int64})
df1 = pl.LazyFrame({"a": [1, 2, 3]})
df2 = pl.concat([df0, df1])

df2.sink_parquet(file)
df3 = pl.read_parquet(file)

assert_frame_equal(df2.collect(), df3)

Log output

Case 1

UNION: union is run in parallel
Traceback (most recent call last):
  File "/home/sson/python/bmf/tmp/gurd/plconcatissue.py", line 9, in <module>
    assert_frame_equal(df2, df2.collect().lazy())
  File "/home/sson/.cache/pypoetry/virtualenvs/foo-hJsE43dC-py3.11/lib/python3.11/site-packages/polars/_utils/deprecation.py", line 91, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sson/.cache/pypoetry/virtualenvs/foo-hJsE43dC-py3.11/lib/python3.11/site-packages/polars/testing/asserts/frame.py", line 91, in assert_frame_equal
    _assert_frame_schema_equal(
  File "/home/sson/.cache/pypoetry/virtualenvs/foo-hJsE43dC-py3.11/lib/python3.11/site-packages/polars/testing/asserts/frame.py", line 175, in _assert_frame_schema_equal
    raise AssertionError(msg)
AssertionError: columns ['a'] in right LazyFrame, but not in left

Case 2

RUN STREAMING PIPELINE
[df -> union -> parquet_sink, df -> union -> parquet_sink]
UNION: union is run in parallel
Traceback (most recent call last):
  File "/home/sson/python/bmf/tmp/gurd/plconcatissue.py", line 14, in <module>
    assert_frame_equal(df2, df3)
  File "/home/sson/.cache/pypoetry/virtualenvs/foo-hJsE43dC-py3.11/lib/python3.11/site-packages/polars/_utils/deprecation.py", line 91, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sson/.cache/pypoetry/virtualenvs/foo-hJsE43dC-py3.11/lib/python3.11/site-packages/polars/testing/asserts/frame.py", line 104, in assert_frame_equal
    raise_assertion_error(
  File "/home/sson/.cache/pypoetry/virtualenvs/foo-hJsE43dC-py3.11/lib/python3.11/site-packages/polars/testing/asserts/utils.py", line 12, in raise_assertion_error
    raise AssertionError(msg) from cause
AssertionError: LazyFrames are different (number of rows does not match)
[left]:  3
[right]: 0

Case 3

RUN STREAMING PIPELINE
[df -> union -> parquet_sink, df -> union -> parquet_sink]
UNION: union is run in parallel
Traceback (most recent call last):
  File "/home/sson/.cache/pypoetry/virtualenvs/foo-hJsE43dC-py3.11/lib/python3.11/site-packages/polars/testing/asserts/frame.py", line 114, in assert_frame_equal
    _assert_series_values_equal(
  File "/home/sson/.cache/pypoetry/virtualenvs/foo-hJsE43dC-py3.11/lib/python3.11/site-packages/polars/testing/asserts/series.py", line 182, in _assert_series_values_equal
    raise_assertion_error(
  File "/home/sson/.cache/pypoetry/virtualenvs/foo-hJsE43dC-py3.11/lib/python3.11/site-packages/polars/testing/asserts/utils.py", line 12, in raise_assertion_error
    raise AssertionError(msg) from cause
AssertionError: Series are different (exact value mismatch)
[left]:  [1, 2, 3]
[right]: [2, 3, 1]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/sson/python/bmf/tmp/gurd/plconcatissue.py", line 14, in <module>
    assert_frame_equal(df2.collect(), df3)
  File "/home/sson/.cache/pypoetry/virtualenvs/foo-hJsE43dC-py3.11/lib/python3.11/site-packages/polars/_utils/deprecation.py", line 91, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sson/.cache/pypoetry/virtualenvs/foo-hJsE43dC-py3.11/lib/python3.11/site-packages/polars/testing/asserts/frame.py", line 124, in assert_frame_equal
    raise_assertion_error(
  File "/home/sson/.cache/pypoetry/virtualenvs/foo-hJsE43dC-py3.11/lib/python3.11/site-packages/polars/testing/asserts/utils.py", line 12, in raise_assertion_error
    raise AssertionError(msg) from cause
AssertionError: DataFrames are different (value mismatch for column 'a')
[left]:  [1, 2, 3]
[right]: [2, 3, 1]

Issue description

The comparisons shouldn't fail as the data/lazyframes should quite clearly be identical.

Expected behavior

All the comparisons should succeed.

Installed versions

--------Version info---------
Polars:              1.6.0
Index type:          UInt32
Platform:            Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.39
Python:              3.11.9 (main, Apr 27 2024, 21:16:11) [GCC 13.2.0]

----Optional dependencies----
adbc_driver_manager  <not installed>
altair               <not installed>
cloudpickle          3.0.0
connectorx           <not installed>
deltalake            <not installed>
fastexcel            <not installed>
fsspec               2024.6.1
gevent               <not installed>
great_tables         <not installed>
matplotlib           <not installed>
nest_asyncio         <not installed>
numpy                2.0.1
openpyxl             3.1.5
pandas               2.2.2
pyarrow              16.1.0
pydantic             2.8.2
pyiceberg            <not installed>
sqlalchemy           2.0.32
torch                <not installed>
xlsx2csv             0.8.3
xlsxwriter           3.2.0

[orlp]

The first two cases is expected behavior, the third case is a bug. @coastalwhite could you check that third case out? For some reason concatenating with an empty LazyFrame screws up the order in sink_parquet.

[Unigurd]

The first two cases were very unexpected to me. Can you explain them or point me to a resource that explains them? I'll elaborate on my thoughts too:

Regarding the first case, I'm not comparing df1 and df2. I'm comparing df2 and df2.collect().lazy(). To me it seems that if you collect a dataframe and make it lazy again, you're left with the original lazyframe. It should be equivalent to the identity operation. If there's an error, it should happen when you call .collect()on the LazyFrame, not when you compare it to the original lazyframe. I don't think it's unlikely in a big chain of computations on a lazyframe that someone's gonna call.collect()somewhere and then.lazy()again somewhere else, and I don't think they're gonna expect their frame to be different because of it. Also, my intuition of what the correct behavior is has been guided byDataFrame`s, so when I saw that

df0 = pl.DataFrame()
df1 = pl.DataFrame({"a": []})
df2 = pl.concat([df0, df1])

yielded a concatenated DataFrame no problem, I figured there was a problem in first example case. I'd still argue that if my first example is expected, then this snippet here is unexpected.

Regarding the second case, I again think that writing a LazyFrame and the reading it again should be the identity operation. It's essentially the same argument as for the first case. Here's an alternate formulation of the second example where I concatenate DataFrames and then make it lazy, and there's no problem here:

df0 = pl.DataFrame()
df1 = pl.DataFrame({"a": [1, 2, 3]})
df2 = pl.concat([df0, df1]).lazy()

df2.sink_parquet(file)
df3 = pl.scan_parquet(file)

assert_frame_equal(df2, df3)

I'd argue that either both this and example 2 should fail, or both should succeed.

I have now clarified what I see as the issue. If this clears it up and cases 1 and/or 2 indeed are issues, then I apologize for not adding this to the original issue description. I simply didn't think to.

[coastalwhite]

I am looking into the 3rd case. I will discuss the other cases with @orlp and come back.

[cmdlineluser]

Should these not produce the same result?

>>> pl.concat([df0, df1]).collect_schema()
# Schema()

>>> pl.concat([df1, df0]).collect_schema()
# Schema([('a', Null)])

[ritchie46]

The last case is due to the parquet sink not being order preserving. This will be fixed in the new streaming engine.

[ritchie46]

This is fixed by the new streaming engine.