chispa icon indicating copy to clipboard operation
chispa copied to clipboard

fixed structfield comparison when dataType is array

Open dfarren opened this issue 4 years ago • 1 comments

The [newly created] test below fails. This because are_structfields_equal doesn't check for the case when the dataType is an array. If the dataType is array, then the nullability shouldn't matter when argument ignore_nullability=True.

def it_can_perform_nullability_insensitive_comparisons_with_arrays():
    sf1 = StructField("hi", ArrayType(IntegerType(), True), False)
    sf2 = StructField("hi", ArrayType(IntegerType(), False), True)
    assert are_structfields_equal(sf1, sf2, ignore_nullability=True) == True

This PR fixes this by adding check_type_equal_ignore_nullable(sf1, sf2) in the are_structfields_equal function.

dfarren avatar Jun 22 '21 16:06 dfarren

Does this work for nested array columns?

MrPowers avatar Sep 30 '23 01:09 MrPowers