kerchunk icon indicating copy to clipboard operation
kerchunk copied to clipboard

example of auto_dask writing to a parquet for for the multi references

Open raybellwaves opened this issue 10 months ago • 1 comments

I'm looking for a combination of https://github.com/ProjectPythia/kerchunk-cookbook/blob/main/notebooks/advanced/Parquet_Reference_Storage.ipynb and https://github.com/ProjectPythia/kerchunk-cookbook/pull/59

and have auto_dask write to a parquet file

raybellwaves avatar Apr 05 '24 03:04 raybellwaves

That's a good point! I think it would need a little work - the first stage tasks don't write their references to files at all, but pass them as data. The final output as parquet would be easier to implement but still need some work, since the same arguments to MZZ are used for the first stage and the final.

martindurant avatar Apr 05 '24 13:04 martindurant