Deepak Cherian
Deepak Cherian
hehhe funnily it seems like you could reverse the graph so inputs become outputs, apply a selection to the inputs to isolate the chunks that have changed, call cull, then...
cc @Mikejmnez
> whether we should continue allowing IndexVariable data be updated in place via .data property. IMO we should really deprecate it I agree. We don't allow it on `.values` anyway...
> I think a big difference is that this "map" approach assumes more knowledge the structure of the dataset up front. :+1: > IIUC having each worker write a region...
I've been thinking that a "CropCodec" might work for uniform overlaps. Here is an example of a TIFF dataset with no overlaps if anyone is interested: https://icechunk.io/en/latest/ingestion/glad-ingest/
> satellite swathes that produce slightly differently shaped data How will this work, since each chunk will require a different padding (presumably)? Are you going to add a new codec?...
> I agree that if the chunksizes are actually (1,1), (1,2), (1,3) then it's a lot harder. IIRC this is a good model for the swath problem. I think the...
> I guess in this case even variable-length chunks wouldn't save us because this would actually require "ragged" chunks Yes exactly. > my understanding of your suggestion is to add...
Can you make this change for `to_dask_dataframe` too please?
haha I knew this was an issue with model output (variables can appear and disappear) but it's news to me that it can happen with nominally-standard-archival data. FWIW I think...