polars icon indicating copy to clipboard operation
polars copied to clipboard

fix: Recurse on map field during type conversion

Open cgbur opened this issue 1 year ago • 2 comments

fixes #14639

When encountering a map column (which in Arrow is a List<Struct<K, V>>), the code failed to recursively convert subtypes into valid Polars types. This caused an error when a map column contained a string (UTF-8) type, as it would not convert the UTF-8 column into a UTF-8 view type, leading to type assertion failures later.

cgbur avatar Mar 14 '24 19:03 cgbur

I have run the make test and make pre-commit as well as verify the changes by hand in parsing parquet. I am not sure why the code coverage check is failing in this case.

Note that when running make test-all, it was failing before and after my change for some number of tests (31 before my change, 31 after). I would appreciate some help or feedback in figuring out what to do to resolve this.

edit:

I've tested again locally, reverting all my changes, and I have test failures and cannot reproduce any of the failures happening in the CodeCov.

cgbur avatar Mar 14 '24 20:03 cgbur

@ritchie46 Sorry to directly tag you, but I think this issue is important. Right now Polars is unable to query parquet files with map columns including Utf8 parquet types. I don't know if this pull request is the right solution, but I think you will know what the issue is.

cgbur avatar Apr 01 '24 16:04 cgbur