kerchunk
kerchunk copied to clipboard
example of auto_dask writing to a parquet for for the multi references
I'm looking for a combination of https://github.com/ProjectPythia/kerchunk-cookbook/blob/main/notebooks/advanced/Parquet_Reference_Storage.ipynb and https://github.com/ProjectPythia/kerchunk-cookbook/pull/59
and have auto_dask
write to a parquet file
That's a good point! I think it would need a little work - the first stage tasks don't write their references to files at all, but pass them as data. The final output as parquet would be easier to implement but still need some work, since the same arguments to MZZ are used for the first stage and the final.