pangeo-forge-recipes icon indicating copy to clipboard operation
pangeo-forge-recipes copied to clipboard

Disambiguate naming of `store_chunk`/`iter_chunks` from `target_chunks`

Open cisaacstern opened this issue 3 years ago • 4 comments

XarrayZarrRecipe uses variations of the name chunks to refer to two different recipe attributes. iter_chunks is a generator which yields inputs for the store_chunk execution step. target_chunks is a dictionary defining zarr chunk sizes. It may be worth considering re-naming iter_chunks (and any/all associated recipe attributes) to make it clear that this method is not related to zarr chunks.

cisaacstern avatar Aug 16 '21 21:08 cisaacstern

Possible alternate names

  • fragments
  • shards
  • blocks
  • segments
  • write fragments / blocks / shards / segments

rabernat avatar Aug 16 '21 21:08 rabernat

fragments shards blocks segments

shards or blocks are appealing to me simply for being shorter, and between them, shards seems less connotative of byte sequences within a file. so maybe shards?

cisaacstern avatar Aug 16 '21 22:08 cisaacstern

Would like to add an endorsement for following the target_chunks naming pattern (<adjective>_<noun>) in the renaming of store_chunk/iter_chunks. Naming in storage.py defines a source/target relationship, so perhaps store_chunk could become something like ... write_source_shard ? Which would make iter_chunks ... iter_source_shards.

cisaacstern avatar Aug 16 '21 23:08 cisaacstern

For anyone looking to grok the somewhat hard-to-grasp notion of the "source chunks" (under discussion for renaming here), reading https://github.com/pangeo-forge/pangeo-forge-recipes/pull/205#issuecomment-923200879 was an "a-ha" moment for me.

cisaacstern avatar Sep 20 '21 19:09 cisaacstern

This issue relates to the pre-beam-refactor XarrayZarrRecipe which we are no longer developing, so closing. As an aside, I think the current StoreToZarr naming does a good job of addressing this prior naming overload, by distinguishing between target_chunks (i.e. zarr chunks), and fragments (which are used to create zarr chunks, but are not 1:1 with zarr chunks).

cisaacstern avatar Aug 24 '23 22:08 cisaacstern