Tom Augspurger
Tom Augspurger
`finalize_target`, specifically https://github.com/pangeo-forge/pangeo-forge-recipes/blob/4f8c0522c1395aaf15a820cfd637a1f1e76729b2/pangeo_forge_recipes/recipes/xarray_zarr.py#L558, can be slow. We call `set(group)` to figure out what which variables are available. I didn't realize it, but apparently that can trigger a full listing of...
Closes https://github.com/pangeo-forge/pangeo-forge-recipes/issues/227 (maybe, might be worth testing). A few notes: 1. I'm not really happy with how I'm handling the locking (see https://github.com/pangeo-forge/pangeo-forge-recipes/pull/229/files#diff-e12c886cc124886c5cfa5313d760a36c39649af9da845077c663e6feab8487b5R245-R253). I just kind of overwrite what the...
Dropping a few notes here for the day: @sharkinsspatial and I are seeing some slowness and high memory usage on the client when running the imerg recipe. This has lots...
See https://github.com/pangeo-forge/pangeo-forge-recipes/pull/160#issuecomment-864075050 for justification, but the tl/dr is that for large recipes (many inputs), `filepatterns_from_sequence` can make large function objects. So while `filepatterns_from_sequence` is very convenient, I think we're best...
This is a followup to https://github.com/pangeo-forge/pangeo-forge/issues/84#issuecomment-812116688. The pangeo-forge library and bakeries need to coordinate on producing and collecting rich logging information for debugging. As recommended in https://docs.python.org/3/howto/logging.html#configuring-logging-for-a-library, libraries like pangeo-forge...
I'm trying to push some data through pangeo-forge-recipes, and local testing is working fine. When I go to write to blob storage, I'm running into a convoluted set of bugs...
This adds a new keyword to AzureBlobFileSystem to limit the number of concurrent connectiouns. See https://github.com/pangeo-forge/pangeo-forge-recipes/issues/227#issuecomment-964259201 for some motivation. In that situation, we had a single FileSystem instance that was...
Ideally we wouldn't have a top-level import from this private module. azure.storage.blob removing that would cause a failure at import time in adlfs. We should see why that's needed and...
This adds a "pc" fsspec filesystem implementation, which lets us insert "pc::" in an fsspec URL and automatically sign it when loading it with an fsspec client. The primary motivation...