polars icon indicating copy to clipboard operation
polars copied to clipboard

Panic on data containing `Decimal("NaN")` and/or `Decimal("Infinity")`

Open alexander-beedie opened this issue 2 years ago • 1 comments

Polars version checks

  • [X] I have checked that this issue has not already been reported.

  • [X] I have confirmed this bug exists on the latest version of Polars.

Issue description

Like float, decimal types also have NaN and Infinity values:

Reproducible example

Succeeds: (float nan/inf)

import polars as pl
s = pl.Series( [float('nan'), float('infinity')] )

# shape: (2,)
# Series: '' [f64]
# [
#    NaN
#    inf
# ]

Panics: (decimal nan/inf)

with pl.Config( activate_decimals=True ):
    s = pl.Series( [Decimal('nan'), Decimal('infinity')] )

# thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: 
#  TypeError("'str' object cannot be interpreted as an integer"),
#   src/conversion.rs:727:72

Notes

There is more than one type of decimal NaN; we should recognise the following strings...

  • NaN, ±NaN, sNAN[^1], ±sNaN

...as well as:

  • Inf, ±Inf, Infinity, ±Infinity

Expected behavior

Should be able to parse/recognise decimal NaN/sNaN, and Inf/Infinity values.

Installed versions

---Version info---
Polars: 0.17.6
Index type: UInt32
Platform: macOS-13.3.1-arm64-arm-64bit
Python: 3.11.3 (main, Apr  7 2023, 20:13:31) [Clang 14.0.0 (clang-1400.0.29.202)]
---Optional dependencies---
numpy: 1.24.2
pandas: 2.0.0
pyarrow: 11.0.0
connectorx: 0.3.2_alpha.2
deltalake: 0.8.1
fsspec: <not installed>
matplotlib: <not installed>
xlsx2csv: 0.8.1
xlsxwriter: 3.0.9

[^1]: sNaN: https://mathworld.wolfram.com/SignalingNaN.html

alexander-beedie avatar Apr 21 '23 19:04 alexander-beedie

How is Nan and infinity stored in a the i128 physical type?

ritchie46 avatar May 30 '23 08:05 ritchie46

Our decimal type is not one-to-one with Python's Decimal type. We currently don't support NaNs/infinities in it, and I don't expect this to ever change, leaving those strictly with the floating point types.

orlp avatar Jan 18 '24 14:01 orlp