pandera icon indicating copy to clipboard operation
pandera copied to clipboard

Support Ibis Backend

Open cosmicBboy opened this issue 1 year ago • 6 comments

Is your feature request related to a problem? Please describe.

Pandera currently doesn't support validating data in a persistent datastore (e.g. MySQL, Postgres, etc). It would benefit users to be able to write pandera schemas that can then be compiled to a query language (like SQL), executed on a remote DB, that either:

  1. validates the data in-place, returning an error report if the data is invalid
  2. validates the data and load it into memory using some framework (e.g. pandas) for further processing

A high-leverage integration to enable this behavior would be with ibis, a data analytics framework that hooks into various backends (duckdb, mysql, postgres, etc).

Describe the solution you'd like

For the MVP integration with ibis:

  • Implement a schema specification for ibis
  • Implement a backend validator for ibis
  • Support writing custom checks

Describe alternatives you've considered NA

cosmicBboy avatar Mar 09 '23 16:03 cosmicBboy