Ritchie Vink

The last case is due to the parquet sink not being order preserving. This will be fixed in the new streaming engine.

This is fixed by the new streaming engine.

Hey @tmct, great that you made an IO plugin! That's where they are for. This example shows the ability to create the schema lazily: https://github.com/pola-rs/pyo3-polars/tree/main/example/io_plugin

@tmct your function can register the callable every `scan_io_function` and on that registration you provided the schema. In pseudo code ```python def scan_foo(file_name) -> pl.LazyFrame: def generator( ) -> Iterator[pl.DataFrame]:...

Right, I agree. It differs from the other scans. It is something that would be problematic for cloud. Is it problematic for your use cases though? Maybe the `schema` should...

Will make it a callable. 👍

That should still work. All scans had the schema in the DSL a year ago and that worked fine for local execution. I am adding the callable option, but I...

Yes, let's expose this to the eager methods a well.

@coastalwhite is this ok?

@henrikig can you use this trait for determining the `unique` values. https://github.com/pola-rs/polars/pull/16765