pandera icon indicating copy to clipboard operation
pandera copied to clipboard

Granular validations on pandas dfs

Open kykyi opened this issue 1 year ago • 0 comments

In an effort to make the error messages produced by an un-rescued SchemaError more insightful at a glance, this PR introduces granular validations on pandas dataframe schemas. This stems from the error report built through the pyspark dataframe schema validations.

In order to follow this behaviour without an extensive re-write, I've tried to map all error reason_codes to either DATA or SCHEMA. If you want to just run data validations or schema validations, this is dictated by your $PANDERA_VALIDATION_DEPTH env var.

This PR takes over from https://github.com/unionai-oss/pandera/pull/1461 and is attempting to resolve this issue: https://github.com/unionai-oss/pandera/issues/1276

Note

This PR is still a WIP, but feel free to comment 😄

kykyi avatar Feb 09 '24 06:02 kykyi