xarray icon indicating copy to clipboard operation
xarray copied to clipboard

drop the length from `numpy`'s fixed-width string dtypes

Open keewis opened this issue 1 year ago • 4 comments

By converting arrays of fixed-width string / bytes dtypes to their base dtype (np.str_ and np.bytes_) in np.result_type, we can avoid accidentally truncating the replacement strings in xr.where.

While this works, I wonder if we instead should ask numpy to do this for us? I.e. np.result_dtype(np.dtype("<U1"), str) should return np.str_, not np.dtype("<U1").

  • [x] Closes #9180
  • [x] Tests added
  • [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

keewis avatar Oct 06 '24 13:10 keewis

While this works, I wonder if we instead should ask numpy to do this for us? I.e. np.result_dtype(np.dtype("<U1"), str) should return np.str_, not np.dtype("<U1").

Yes, this would be better in my opinion!

shoyer avatar Oct 10 '24 09:10 shoyer

how do we proceed, then? Merge this (after fixing the failing min-deps CI), ask if numpy.result_type can be changed, and remove it once we can require a version of numpy that does this for us?

keewis avatar Oct 10 '24 09:10 keewis

Yes, that’s probably the way to go

On Thu, Oct 10, 2024 at 6:51 PM Justus Magin @.***> wrote:

how do we proceed, then? Merge this (after fixing the failing min-deps CI), ask if numpy.result_type can be changed, and remove it once we can require a version of numpy that supports this?

— Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/pull/9586#issuecomment-2404624351, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJJFVSTJCPVMIHB7YF2QKTZ2ZE2LAVCNFSM6AAAAABPOKUAHKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBUGYZDIMZVGE . You are receiving this because you commented.Message ID: @.***>

shoyer avatar Oct 10 '24 13:10 shoyer

Yes, this would be better in my opinion!

There's some concerns about this in numpy/numpy#27546

keewis avatar Oct 15 '24 12:10 keewis

@TomNicholas, should we merge this before the release?

keewis avatar Oct 24 '24 21:10 keewis

Sure! If there is any doubt then leave it, but Stephan reviewed it so I say just merge.

TomNicholas avatar Oct 24 '24 21:10 TomNicholas

the only doubt is about what should happen upstream in numpy (if anything should happen at all), so that shouldn't block us here

keewis avatar Oct 24 '24 21:10 keewis

I agree, let's merge.

TomNicholas avatar Oct 24 '24 21:10 TomNicholas