chispa
chispa copied to clipboard
fixed structfield comparison when dataType is array
The [newly created] test below fails. This because are_structfields_equal doesn't check for the case when the dataType is an array. If the dataType is array, then the nullability shouldn't matter when argument ignore_nullability=True.
def it_can_perform_nullability_insensitive_comparisons_with_arrays():
sf1 = StructField("hi", ArrayType(IntegerType(), True), False)
sf2 = StructField("hi", ArrayType(IntegerType(), False), True)
assert are_structfields_equal(sf1, sf2, ignore_nullability=True) == True
This PR fixes this by adding check_type_equal_ignore_nullable(sf1, sf2) in the are_structfields_equal function.
Does this work for nested array columns?