pandera
pandera copied to clipboard
Granular validations on pandas dfs
In an effort to make the error messages produced by an un-rescued SchemaError
more insightful at a glance, this PR introduces granular validations on pandas dataframe schemas. This stems from the error report built through the pyspark dataframe schema validations.
In order to follow this behaviour without an extensive re-write, I've tried to map all error reason_code
s to either DATA
or SCHEMA
. If you want to just run data validations or schema validations, this is dictated by your $PANDERA_VALIDATION_DEPTH
env var.
This PR takes over from https://github.com/unionai-oss/pandera/pull/1461 and is attempting to resolve this issue: https://github.com/unionai-oss/pandera/issues/1276
Note
This PR is still a WIP, but feel free to comment 😄