narwhals-dev
[Enh]: Pydantic integration
We would like to learn about your use case. For example, if this feature is needed to adopt Narwhals in an open source project, could you please enter the link to it below?
https://anam-org.github.io/metaxy/main/
Please describe the purpose of the new feature or describe the problem to solve.
I would like to convert my Pydantic models into Narwhals schemas. I am probably going to implement something myself right now, but I feel like this feature is general and useful enough to belong to Narwhals itself.
Suggest a solution if possible.
No response
If you have tried alternatives, please describe them below.
Something like https://github.com/FBruzzesi/anyschema, but staying in Narwhals land instead.
Additional information that may help us understand your needs.
I need to make sure DataFrames have the right schema before writing them to storage formats that support schema evolution (such as DeltaLake or LanceDB).
My dataframes are always associated with Pydantic models, so this functionality is exactly what I'd like to have.
Something like https://github.com/FBruzzesi/anyschema, but staying in Narwhals land instead.
I am flattered 🙏🏼
I hinted a few times to the possibility of moving it under the narwhals-dev organization eventually, but the current state is not really usable and I didn't have too much time/motivation to try developing it further
My dataframes are always associated with Pydantic models, so this functionality is exactly what I'd like to have.
Disclaimer: I am not very familiar with its API nor dependencies
Maybe Pandera can help?
- They might eventually support narwhals https://github.com/unionai-oss/pandera/pull/2081
- Maybe there is already a way to use what they built?
Nah, I want a very lightweight, dependency-free solution, just to do schema conversion. No data validation or anything like that. Sounds like this should belong to Narwhals (integration).
@danielgafni I will take a stab at getting anyschema to a somewhat usable state during the weekend. I will start by targeting Pydantic models. Do you mind if I tag you there eventually?
@danielgafni I am going to close this issue as I don't think this feature will end up within the narwhals library itself. Maybe in the narwhals ecosystem 😉