polars
polars copied to clipboard
write_parquet and sink_parquet delete existing file if locked
Checks
- [x] I have checked that this issue has not already been reported.
- [x] I have confirmed this bug exists on the latest version of Polars.
Reproducible example
import polars as pl
from filelock import FileLock
import time
data = {
"col1": [1,2,3],
"col2": [4,5,6],
}
test = pl.DataFrame(data)
test.write_parquet('test.parquet')
time.sleep(2)
test = test.select(pl.col("col1"))
lock = FileLock('test.parquet')
with lock:
test.write_parquet('test.parquet')
Log output
Issue description
Due to the Lock Polars returns a ComputeError and deletes the file, even if it was completely valid.
Expected behavior
The file should not be deleted if polars cannot touch it
Installed versions
--------Version info---------
Polars: 1.35.2
Index type: UInt32
Platform: Windows-10-10.0.22631-SP0
Python: 3.11.9 (tags/v3.11.9:de54cf5, Apr 2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)]
Runtime: rt32
----Optional dependencies----
Azure CLI <not installed>
'az' is not recognized as an internal or external command,
operable program or batch file.
adbc_driver_manager <not installed>
altair <not installed>
azure.identity <not installed>
boto3 <not installed>
cloudpickle <not installed>
connectorx <not installed>
deltalake <not installed>
fastexcel 0.14.0
fsspec <not installed>
gevent <not installed>
google.auth 2.40.3
great_tables <not installed>
matplotlib <not installed>
numpy 2.2.6
openpyxl 3.1.5
pandas 2.2.3
polars_cloud <not installed>
pyarrow 21.0.0
pydantic <not installed>
pyiceberg <not installed>
sqlalchemy <not installed>
torch <not installed>
xlsx2csv <not installed>
xlsxwriter <not installed>
I can't reproduce on MacOS. You didn't follow the issue template and showed us the output of pl.show_versions().
I have added the output of pl.show_versions()