Charles Stern
Charles Stern
@martindurant, with `fsspec 2021.11.1` caching throughput over sftp for this recipe was in the range of `0.06 - 0.1 MB/s`. https://github.com/paramiko/paramiko/issues/175 lead me to try [`f.prefetch()` in `SFTPFileSystem._open`](https://github.com/cisaacstern/filesystem_spec/compare/master...sftp-prefetch#diff-1e992b6bf5e514e63fc8b97f7b98e61acb618b7551ee966b010c800a68073b1b), which increased...
@roxyboy, as of https://github.com/pangeo-data/swot_adac_ogcms/pull/10, the FIO-COM32 datasets are now accessible via our project catalog. I've updated the example notebook to demonstrate accessing these data: https://github.com/pangeo-data/swot_adac_ogcms/blob/main/intake_demo.ipynb The access pattern is the...
@jkingslake, thanks for submitting this request. Assuming we do go with the larger dataset, where do the source files live for that? Apologies if I missed that in your initial...
> I could produce a FTP link with password This would work. We've done something similar before. Out of curiosity, how large is the (larger) Source dataset? In terms of...
Great. This sounds quite manageable. Please let me know when the FTP link is available and we can begin the recipe development.
Great! A description of the recipe development process is given here: https://pangeo-forge.readthedocs.io/en/latest/intro_tutorial.html (This documentation is still quite fresh, so we definitely welcome feedback on it!) As you will see, the...
Based on https://github.com/pangeo-forge/staged-recipes/issues/90#issuecomment-932584463, it seems the high resolution dataset of interest is not yet published. It would be preferable from a provenance and recipe standpoint to build the Zarr store...
> we could convert it to zarr format and store (publish) it somewhere permanently (in the cloud)? Yes, with the exception of the permanent part. While any zarr store we...
> Under what circumstances can we actually just publish the data? Not all contributors will have their data in an existing repository? Should we start an Issue in https://github.com/pangeo-forge/roadmap/issues to...
> Did anyone try the http link? Hi @talbrecht thanks for following up. @jkingslake, is this recipe something you are interested in working on? Pangeo Forge is designed as a...