sarpy icon indicating copy to clipboard operation
sarpy copied to clipboard

Support for S3 bucket

Open effeminati opened this issue 1 year ago • 3 comments

I would like to process data directly on the cloud without the need to download data to the local filesystem. I tried, but I was not able to open remote files with Sarpy.

Do you foresee extending the interface to do this?

Sarpy is a valuable library capable of opening and standardizing data from many different SAR sensors. Making it able to work on the cloud would make a significant difference.

effeminati avatar Jul 08 '24 14:07 effeminati

Yes! As you point out, only a very limited subset of interfaces (e.g.) are architected in a way that is compatible with something like smart-open.

There is currently an effort underway to modernize SarPy, and standardizing I/O in a way that enables working with large remote files is a key goal.

pressler-vsc avatar Jul 08 '24 14:07 pressler-vsc

For the file types that are currently supported you can see an example of how to do so here:

https://sarpy.readthedocs.io/en/latest/examples/first_example.html#opening-remote-file

naething avatar Jul 25 '24 01:07 naething

@dhaverporth should we investigate this enhancement request?

petersontex avatar May 07 '25 15:05 petersontex

@effeminati thank you for your request. Per the above comments this capability is in sarpy.smart_open. Please let us know if you are having any issues with smart_open.

petersontex avatar Jun 18 '25 20:06 petersontex