dask-sql
dask-sql copied to clipboard
[DF] Honor column nullability flag from parquet files
Is your feature request related to a problem? Please describe. When reading tables from Parquet files the "nullability" of the column should be read from the source parquet file. Currently we just hardcode the nullability to false which is inaccurate.
Describe the solution you'd like
When registering a table in the DaskSqlContext the nullability of the columns should be passed along in the schema metadata so that the appropriate calls can be made to Field::new(...)
Describe alternatives you've considered None
Additional context None