spark-expectations icon indicating copy to clipboard operation
spark-expectations copied to clipboard

[BUG] The input_count is less than the output_count in the stats table

Open DevathiNNikhil opened this issue 1 year ago • 1 comments

Describe the bug The values that are observed in the output_count are sometimes more than the value of input_count. This case was happened only few times for the many times the execution has happened

To Reproduce The checks performed on the table are row_dq and to be specific the rules are null_validation and uniqueness check and the rules was defined as it is defined in documentation. All the rules has been passed successfully with zero error rate but out of my experimentation on many tables it specifically happened on 2 tables

Expected behavior Expected that input_counts to be equal to output_counts

Screenshots image001

The spark expectations version that has been used is 0.8.1

DevathiNNikhil avatar Jan 05 '24 13:01 DevathiNNikhil