NamedArray tracking issue

[dcherian]

@andersy005 I think it would be good to keep a running list of NamedArray tasks. I'll start with a rough sketch, please update/edit as you like.

• [x] Refactor out NamedArray base class (#8075)
• [x] publicize design doc: Scientific Python | Pangeo | NumPy Mailist
• [ ] Migrate VariableArithmetic to NamedArrayArithmetic (#8244)
• [ ] Migrate ExplicitlyIndexed array classes to array protocols
• [x] MIgrate from *Indexer objects to .oindex and .vindex on ExplicitlyIndexed array classes
• [x] https://github.com/pydata/xarray/pull/8870
• [ ] Migrate unary ops
• [ ] Migrate binary ops
• [ ] Migrate nanops.py
• [x] Avoid "injecting" reduce methods potentially by using generate_reductions.py? (#8304)
• [ ] reprs and formatting.py
• [x] parallelcompat.py
• [ ] pycompat.py (#8244)
• [ ] https://github.com/pydata/xarray/pull/8276
• [ ] have test_variable.py test both NamedArray and Variable
• [x] Arrays with unknown shape #8291
• [ ] https://github.com/pydata/xarray/issues/8306
• [ ] https://github.com/pydata/xarray/issues/8310
• [ ] https://github.com/pydata/xarray/issues/8333
• [ ] Try to preserve imports from xarray.core/* by importing namedarray functionality into xarray.core/*

xref #3981

[headtr1ck]

For the operators we will run into the following issue:

Currently Variable + DataArray will return a DataArray because Variable will return NotImplemented and then python calls DataArray.__radd__. This is hard coded.

If you want to make NamedArray independent from xarray, you need to find a way to solve this. I assume numpy has the same issues, maybe check over there how it works?

[dcherian]

@headtr1ck or @Illviljan would one of you be able to handle moving over unary and binary ops and maybe even nanops. I think that might involve a bunch of typing wrangling that you two would be best suited to fix

[headtr1ck]

I can have a look at it :) By the way: do we want to get rid of the fastpath argument to the __init__ and replace it with a custom classmethod? I think it is confusing to have this in a public interface.

[Illviljan]

How much should NamedArray conform to the array api standard?

For example, the array api does not implement the methods, .imag, .real, .astype. https://data-apis.org/array-api/latest/API_specification/index.html

I'm leaning towards being strict with the array api and to remove them because they are causing some truly difficult typing issues (#8281, #8294). If all these shape/dim/dtype-changing methods were moved to functions instead the type hinting would be much more straightforward.

[dcherian]

@TomNicholas do you have time to move over parallelcompat.py to namedarray/? It'll be a lot quicker if you did it ;)

[dcherian]

How much should NamedArray conform to the array api standard?

The API is certainly going to be a superset of the array API (that doesn't even have NaN skipping reductions!)

I'm leaning towards being strict with the array api and to remove them because they are causing some truly difficult typing issues

I don't think typing should limit the API we offer.

For example, the array api does not implement the methods, .imag, .real, .astype. data-apis.org/array-api/latest/API_specification/index.html

Can we add these back please? As of now, our plan is to include them. https://github.com/pydata/xarray/blob/main/design_notes/named_array_design_doc.md#attributes-to-be-preserved-from-xarrayvariable

[TomNicholas]

The API is certainly going to be a superset of the array API (that doesn't even have NaN skipping reductions!)

We should also remember that the array API is not fixed in stone - it can and already has changed to include more functions. We should be raising issues there to get functions we want added. See https://github.com/data-apis/array-api/issues/187#issuecomment-1553615779 and https://github.com/data-apis/array-api/issues/669 for example.

I don't think typing should limit the API we offer.

+1

TomNicholas do you have time to move over parallelcompat.py to namedarray/? It'll be a lot quicker if you did it ;)

I'll have a look now!

[headtr1ck]

We should slow down with the PRs, haha. The merge conflicts will be a nightmare

[kgryte]

Hey all! Don't want to hijack this thread, but wanted to respond to a few of the comments above regarding the Array API.

For example, the array api does not implement the methods, .imag, .real, .astype. data-apis.org/array-api/latest/API_specification/index.html

@Illviljan In the Array API standard, we've preferred functional APIs over methods and have the equivalent of imag(), real(), and astype().

The API is certainly going to be a superset of the array API (that doesn't even have NaN skipping reductions!)

@dcherian There has been some discussion about adding support for NaN reductions to the specification (see https://github.com/data-apis/array-api/issues/621); however, there is still some debate as to what form that should take. E.g., whether we actually want nan* variants, a single NaN reduction wrapper, or kwarg support to existing reductions. You're encouraged to participate in that thread to help provide input regarding what makes the most sense from your POV.

We should also remember that the array API is not fixed in stone - it can and already has changed to include more functions. We should be raising issues there to get functions we want added.

@TomNicholas Indeed! May be good to list all the current Array API pain points (if not done already), so that we can potentially prioritize. We could also potentially arrange for xarray devs to present at one of the Consortium workgroup meetings.

Regardless, thanks for considering the Array API standard. We'd love to get additional feedback on the specification over on the specification repo. :)

[maxrjones]

@andersy005 will there be a function similar to as_compatible_data in NamedArray? I'm asking because this affects what tests would be expected to pass/fail in NamedArray vs. Variable (xref https://github.com/pydata/xarray/issues/8370). For example, as_compatible_data currently converts lists to numpy arrays such that https://github.com/pydata/xarray/blob/72abfdf9bd6aef99c0397f6dadc383c28abc6ce0/xarray/tests/test_namedarray.py#L73-L75 raises an AttributeError for NamedArray but not Variable. It would be helpful to better understand whether this will always be the case.

[andersy005]

@andersy005 will there be a function similar to as_compatible_data in NamedArray? I'm asking because this affects what tests would be expected to pass/fail in NamedArray vs. Variable (xref #8370). For example, as_compatible_data currently converts lists to numpy arrays such that

@maxrjones, currently, the constructor of NamedArray.__init__() strictly requires passing an array-like object but we have had discussions regarding reintroducing the as_compatible_data function in the NamedArray.__init__() method, along with the NamedArray.from_array() method. .from_array() method would allow for skipping certain compatibility checks and normalization steps performed in the as_compatible_data() function: https://github.com/pydata/xarray/pull/8294#discussion_r1365843106.

[maxrjones]

@andersy005 will there be a function similar to as_compatible_data in NamedArray? I'm asking because this affects what tests would be expected to pass/fail in NamedArray vs. Variable (xref #8370). For example, as_compatible_data currently converts lists to numpy arrays such that

@maxrjones, currently, the constructor of NamedArray.__init__() strictly requires passing an array-like object but we have had discussions regarding reintroducing the as_compatible_data function in the NamedArray.__init__() method, along with the NamedArray.from_array() method. .from_array() method would allow for skipping certain compatibility checks and normalization steps performed in the as_compatible_data() function: #8294 (comment).

This is helpful, thank you! I will proceed by marking these as xfail for now, so it would be easy to test in NamedArray if added in the future.