RMS icon indicating copy to clipboard operation
RMS copied to clipboard

Fix byte/int8 overflow with numpy >= 2.0

Open adalava opened this issue 1 year ago • 3 comments

** I used to request merging into "dev" branch but it seems to be stale - please change if this is the incorrect branch

Before numpy 2.0 a value of 255 was automatically converted to -1 for int8/byte data type. Last numpy versions from 1.x series show "deprecation warning" but after 2.x it causes a overflow exception and application crashes.

This change converts data to a larger data size before converting it back to int8/byte. This should give same result.

I was not able to test it on s RMS/jessie image, but I tested the approach on python 2.7. See bellow the results


Python 2.7.18 (default, Apr 20 2020, 19:27:10) 
[GCC 8.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> (np.array(np.array([[0, -128], [-100, 100], [37, -80], [255, 0]], np.byte))*255).astype(np.byte) # deprecation warning
array([[   0, -128],
       [ 100, -100],
       [ -37,   80],
       [   1,    0]], dtype=int8)
>>> 
>>> 
>>> (np.array(np.array([[0, -128], [-100, 100], [37, -80], [255, 0]], np.int16)).astype(np.byte)*255).astype(np.byte) # ok
array([[   0, -128],
       [ 100, -100],
       [ -37,   80],
       [   1,    0]], dtype=int8)
>>> 

Python 3.13.0 (main, Oct  8 2024, 00:00:00) [GCC 14.2.1 20240912 (Red Hat 14.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> (np.array(np.array([[0, -128], [-100, 100], [37, -80], [255, 0]], np.byte))*255).astype(np.byte) # deprecation warning
<python-input-1>:1: DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays.  The conversion of 255 to int8 will fail in the future.
For the old behavior, usually:
    np.array(value).astype(dtype)
will give the desired result (the cast overflows).
  (np.array(np.array([[0, -128], [-100, 100], [37, -80], [255, 0]], np.byte))*255).astype(np.byte) # deprecation warning
array([[   0, -128],
       [ 100, -100],
       [ -37,   80],
       [   1,    0]], dtype=int8)
>>> 
>>> (np.array(np.array([[0, -128], [-100, 100], [37, -80], [255, 0]], np.int16)).astype(np.byte)*255).astype(np.byte) # ok
array([[   0, -128],
       [ 100, -100],
       [ -37,   80],
       [   1,    0]], dtype=int8)
>>> 

adalava avatar Nov 18 '24 23:11 adalava

I was wondering if instead of:

img_flatten = (img.flatten().astype(np.byte).astype(np.int16)*255).astype(np.byte)

it should be:

img_flatten = (img.flatten()*255).astype(np.byte)

But results could be different than today's....

adalava avatar Nov 19 '24 05:11 adalava

@dvida , gentle ping just to make sure you are aware of this PR. I'm seeing many stations requesting numpy =>2.0 despite RMS expliciting selecting the 1.x series.

adalava avatar Dec 17 '24 23:12 adalava

Could you try it with this code?

img_flatten = (img.ravel() * 255).astype(np.uint8).view(np.int8)

dvida avatar Jun 08 '25 22:06 dvida