delta-rs
delta-rs copied to clipboard
Impossible to append to a DeltaTable with float data type on RHEL
Environment
Delta-rs version: 0.17.4
Binding: Python 3.11.5
Environment: OS: RHEL 9.3 or WSL2 (Ubuntu 22.04)
Bug
What happened: I created a DeltaTable where most of the columns have a float type. I tried to append a polars DataFrame to the DeltaTable but I got the following error by setting RUST_BACKTRACE=full
thread '<unnamed>' panicked at python/src/lib.rs:1513:48:
called `Result::unwrap()` on an `Err` value: CDataInterface("The datatype \"Float32\" expects 2 buffers, but requested 2. Please verify that the C data interface is correctly implemented.")
stack backtrace:
0: 0x7f000d385ba2 - <unknown>
1: 0x7f000d3b696c - <unknown>
2: 0x7f000d3824df - <unknown>
3: 0x7f000d385974 - <unknown>
4: 0x7f000d38712b - <unknown>
5: 0x7f000d386e83 - <unknown>
6: 0x7f000d3875cd - <unknown>
7: 0x7f000d3874a2 - <unknown>
8: 0x7f000d386076 - <unknown>
9: 0x7f000d3871d4 - <unknown>
10: 0x7f0009fd27d5 - <unknown>
11: 0x7f0009fd2cd3 - <unknown>
12: 0x7f000a01d860 - <unknown>
13: 0x7f000a0c0c02 - <unknown>
14: 0x7f000a1bc5d6 - <unknown>
15: 0x7f000a1bd5f3 - <unknown>
16: 0x7f000a18fbfa - <unknown>
17: 0x7f000a1bc781 - <unknown>
18: 0x7f00151d44a2 - <unknown>
19: 0x7f00151b50c6 - _PyObject_MakeTpCall
20: 0x7f00151bdd26 - _PyEval_EvalFrameDefault
21: 0x7f00151b9c52 - <unknown>
22: 0x7f00151e5c37 - <unknown>
23: 0x7f00151c2236 - _PyEval_EvalFrameDefault
24: 0x7f00151b9c52 - <unknown>
25: 0x7f0015245ff6 - PyEval_EvalCode
26: 0x7f0015264994 - <unknown>
27: 0x7f0015260e16 - <unknown>
28: 0x7f001519e172 - <unknown>
29: 0x7f001519e31b - _PyRun_InteractiveLoopObject
30: 0x7f00151282cd - <unknown>
31: 0x7f001519e4a4 - PyRun_AnyFileExFlags
32: 0x7f0015123e4a - <unknown>
33: 0x7f0015233bad - Py_BytesMain
34: 0x7f0014c29590 - __libc_start_call_main
35: 0x7f0014c29640 - __libc_start_main_alias_1
36: 0x563746731095 - _start
37: 0x0 - <unknown>
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/airflow/.cache/pypoetry/virtualenvs/dataflow-tK0vRoRm-py3.11/lib/python3.11/site-packages/datadex/table.py", line 145, in append
data.write_delta(
File "/home/airflow/.cache/pypoetry/virtualenvs/dataflow-tK0vRoRm-py3.11/lib64/python3.11/site-packages/polars/dataframe/frame.py", line 3589, in write_delta
write_deltalake(
File "/home/airflow/.cache/pypoetry/virtualenvs/dataflow-tK0vRoRm-py3.11/lib64/python3.11/site-packages/deltalake/writer.py", line 319, in write_deltalake
write_deltalake_rust(
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: CDataInterface("The datatype \"Float32\" expects 2 buffers, but requested 2. Please verify that the C data interface is correctly implemented.")
What you expected to happen: I can append to a DeltaTable which is previously created.