ibis
ibis copied to clipboard
feat: allow specify the length of string column
Is your feature request related to a problem?
We are using Snowflake backend, and trying to load parquet file into the snowflake, the parquet has columns with fixed length string type, but after read_parquet called, all column became VARCHAR(16777216), tried other backend, like duckdb/mysql, all seems to upscale the string type to max string length instead of maintaining the length from source. ideally this should be presered.
What is the motivation behind your request?
Our data governance require some strict data type checking, and preserving the source string length will be greatly valuable.
Describe the solution you'd like
I would like to extend ibis schema to allow specify max length of a string column, and also make read_parquet to honor the fixed length physical type.
What version of ibis are you running?
9.0.0
What backend(s) are you using, if any?
MySQL, MSSQL, Snowflake
Code of Conduct
- [X] I agree to follow this project's Code of Conduct