xarray Representing & checking Dataset schemas

What would be the best way to canonically describe a dataset, which could be read by both humans and machines?

For example, frequently in our code we have docstrings which look something like:

def get_returns(security_ids):
    """
    Retuns mega-dimensional dataset which gives recent returns for a set of
        securities by:
    - Date
    - Return (raw / economic / smoothed / etc)
    - Scaling (constant / risk_scaled)
    - Span
    - Hedged vs Unhedged

    Dataset keys are security ids. All dimensions have coords.
    """

This helps when attempting to understand what code is doing while only reading it. But this isn't consistent between docstrings and can't be read or checked by a machine. Has anyone solved this problem / have any suggestions for resources out there?

Tangentially related to https://github.com/python/typing/issues/513 (but our issues are less about the type, dimension sizes, and more about the arrays within a dataset, their dimensions, and their names)

Feb 09 '18 18:02 max-sixty

I think the right word for this may be "schema". For applications and models (rather than data analysis), these sort of conventions can be super-valuable. I like the idea of declarative spec that can be validated.

Just googling around, I came up with pandas-validator: https://github.com/c-data/pandas-validator

Feb 09 '18 19:02 shoyer

I think the right word for this may be "schema"

Right! 🤦‍♂️

Just googling around, I came up with pandas-validator

Interesting, thanks.

Do you think this fits into a 'function which validates', rather than a Mypy-like type annotation? I think ideally there would be a representation of the schema that could work with both, so maybe this isn't the important question atm.

Feb 09 '18 22:02 max-sixty

And let me know if there are already textual schema definitions from other libraries that you think are good, before we go and build our own (we don't work with any netCDF-like files so don't have that context)

Feb 09 '18 23:02 max-sixty

ncdump -h (xarray.Dataset.info()) is one existing schema of sorts, but it's hardly machine readable.

Feb 09 '18 23:02 shoyer

Somewhat related to this issue, I have implemented in xarray-simlab some logic to validate xarray.Variable objects (dimensions, dtype, etc.). See this base class and some sub-classes here. I use that in a way which is quite similar to pandas-validator(i.e., using class attributes).

I'm currently in the process of refactoring this using attrs, which supports both validator functions and type annotations. Not sure how to use the latter for xarray objects, though (BTW I wasn't aware of python/typing#513, good to know!!).

I agree that it would be nice to have a more generic way to describe xarray objects that can be reused in many contexts.

Feb 22 '18 23:02 benbovy

@benbovy That looks v interesting. I think at the moment it would require a bit of work to validate normal xarray objects, is that right? (I'm looking at the __init__, which doesn't take the traditional args supplied to a Variable - tell me if I'm misreading it)

Separately - I didn't know about the project but looks awesome. Do we have a list of projects that integrate xarray? Let's start one somewhere if not @pydata/xarray ?

Feb 22 '18 23:02 max-sixty

@maxim-lian you're right. In this case xsimlab.Variable is a different concept than xarray.Variable, despite that they both have the same name. The former is tight to the modelling framework while the latter is only used for simulation inputs and outputs in xarray-simlab.

Do we have a list of projects that integrate xarray?

There is an ongoing discussion in #1850 about having something like xarray-contrib (likely a github organization).

Feb 22 '18 23:02 benbovy

The commentary in https://github.com/python/typing/issues/513, and @shoyer 's doc https://docs.google.com/document/d/1vpMse4c6DrWH5rq2tQSx3qwP_m_0lyn-Ij4WHqQqRHY/edit#heading=h.rkj7d39awayl are good & growing

I'll close this as I think riding on those coattails - with the addition of names and Datasets as containers - makes the most sense.

(though reopen if we think there's something we could productively do separately)

Mar 28 '18 19:03 max-sixty

I'm really interested in a machine-readable schema for xarray!

Pandera provides machine-readable schemas for Pandas and, as of version 0.7, panderas has decoupled pandera and pandas types to make pandera more useful for things like xarray. I haven't tried pandera yet but I plan to do some experiments soon.

Oct 08 '21 07:10 JackKelly

Pandera provides machine-readable schemas for Pandas and, as of version 0.7, panderas has decoupled pandera and pandas types to make pandera more useful for things like xarray. I haven't tried pandera yet but I plan to do some experiments soon.

Awesome -- would love to hear how this goes!

Oct 08 '21 07:10 shoyer

OK, I think pandera isn't the way forwards because it appears very tighly coupled to Pandas (so, for example, I don't think it's possible to use pandera with n-dimensional arrays).

But Pydantic looks promising. Here's a very quick coding experiment showing one way to use pydantic with xarray... it validates a few things; but it's not super-useful as a human-readable specification for what's going on inside a DataArray or Dataset.

Oct 08 '21 14:10 JackKelly