spatialpandas icon indicating copy to clipboard operation
spatialpandas copied to clipboard

`pack_partitions_to_parquet` hangs silently if destination already exists and `overwrite=False`

Open ahnsws opened this issue 9 months ago • 0 comments

Hello, it looks like if I write to a parquet folder using pack_partitions_to_parquet, it runs successfully once, but upon running it again, if overwrite is False, it silently hangs. The script is below:

import dask.dataframe as dd
import spatialpandas as spd
from spatialpandas.dask import DaskGeoDataFrame

points = spd.geometry.PointArray([[0, 0], [1, 1]])
sdf = spd.GeoDataFrame({"geometry": points})
sddf = dd.from_pandas(sdf, npartitions=2)
assert isinstance(sddf, DaskGeoDataFrame)

sddf.pack_partitions_to_parquet("data.parq")
sddf.pack_partitions_to_parquet("data.parq")  # hangs silently

My environment is:

OS: Ubuntu 22.04.5 LTS
python 3.11
dask 2025.3
distributed 2025.3
spatialpandas 0.5.0

ahnsws avatar Apr 16 '25 23:04 ahnsws