pandera
pandera copied to clipboard
Support Ibis Backend
Is your feature request related to a problem? Please describe.
Pandera currently doesn't support validating data in a persistent datastore (e.g. MySQL, Postgres, etc). It would benefit users to be able to write pandera schemas that can then be compiled to a query language (like SQL), executed on a remote DB, that either:
- validates the data in-place, returning an error report if the data is invalid
- validates the data and load it into memory using some framework (e.g. pandas) for further processing
A high-leverage integration to enable this behavior would be with ibis, a data analytics framework that hooks into various backends (duckdb, mysql, postgres, etc).
Describe the solution you'd like
For the MVP integration with ibis:
- Implement a schema specification for ibis
- Implement a backend validator for ibis
- Support writing custom checks
Describe alternatives you've considered NA