Hugo Pendlebury

icon location Data engineer with an interest in nature.

Results 2 comments of Hugo Pendlebury

Support HDF5 reading/writing (with optional dependencies)

> I'm really excited about Polars. But almost all of my data is in large HDF5 files (actually, NetCDF). I can convert to Parquet files. But loading directly from HDF5...

`sink_parquet(...)` much slower than `.collect().write_parquet(...)`

> @itamarst > > ``` > In [10]: %timeit -n1 -r1 pl.scan_parquet("tmp/pl-perf-medium.parquet").sink_parquet("/tmp/xx", compression="uncompressed") > 2.29 s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop...