spiraldb
LayoutWriter doesn't respect validity of the top level Struct array
Right now if StructArray has top level validity that's not AllValid or NonNullable the validity is discarded
Could you please provide more details? I'm interested in this issue and would like to attempt to resolve it.
This is likely not a good first issue. The solution here is likely to either forbid writing top level structs with validity other than nonnullable and provide a function to push validity of the struct to the children, i.e. run a filter. Alternatively we need to teach column layout about one extra validity array which shouldn't be bad but likely unnecessary
In the current format, isn't it true that this is actually any struct array? I'd like to at least known the fix here to make sure it's not a breaking change when we do.
We should assert at write-time that struct arrays have Validity::NonNullable or Validity::AllValid