Add support for ASF-extracted burst SLC products

[forrestfwilliams]

ASF has been working to provide users the capability to download single-burst Sentinel-1 SLC data. This PR adds support for these products to s1-reader, allowing users to load an ASF-extracted burst SLC product as a Sentinel1BurstSlc object.

To add this functionality, I've made the following modifications:

• s1_reader.load_from_xml was split into two functions; one function that loads data lxml objects (load_from_xml), and a new function that parses the data from these objects (_bursts_from_et)
• Modified arguments of s1_reader.get_path_aux_cal so that it takes platform_id and sensing_start as arguments instead of annotation_path
• Added s1_reader.get_subxml_from_burst_metadata which extracts a desired metadata file from a combined XML file
• Added s1_reader.burst_from_combined_xml which creates a Sentinel1BurstSlc object from combined XML data
• Added s1_reader.load_single_burst as a user-facing function for load an ASF-extracted burst
• Modified s1_burst_slc.Sentinel1BurstSlc.slc_to_vrt_file so that it uses lxml to create VRTs and so that it skips the subsetting of the SLC data if it is working with an extracted burst tiff.

I imagine you all will have many code suggestions, formatting updates, and test additions, so please request that I make these changes, or feel free to add them yourself.

[scottstanie]

very cool! Is there an easy way to add a small sample so we can put a test for the new version? I'm not sure how complete of a .SAFE file you need to have in addition to the VRT, but if you need fake small test data, this unfinished PR has a function to make empty tiffs that are only a few kilobytes https://github.com/isce-framework/s1-reader/pull/84/files#diff-310b112fe464b1fb86176be502b66826e3bd4f6b5e937843f78c3f42141e752e (still need to go back and just merge that without zenodo)

[forrestfwilliams]

Hey @scottstanie thanks! I might have to think a bit about how to get you some small sample data. During burst extraction we repackage the metadata into a single XML file and the data into a geotiff. The geotiff should be relatively straightforward with your code, but the metadata could be trickier. If you want to read more about the data format, check the "Products" tab on this page. This documentation is hot of the press, so let us know if there is anything that's unclear, or that should be added.

[scottstanie]

Ok sounds good! My first thought was to download a burst, zip up the metadata (looks like it's only 1-2 MB zipped), then zero out the .tiff file so it compresses.

I tried S1_292315_IW3_20230922T015859_VV_9961-BURST with the code (downloaded the .tiff and VRT)

In [1]: import s1reader.s1_reader
In [2]: s1reader.s1_reader.load_single_burst("S1_292315_IW3_20230922T015859_VV_9961-BURST.tiff", "S1_292315_IW3_20230922T015859_VV_9961-BURST.xml", "S1A_OPER_AUX_POEORB_OPOD_20231012T080638_V20230921T225942_20230923T005942.EOF")

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <module>:1                                                                                    │
│                                                                                                  │
│ /home/scott/repos/s1-reader/src/s1reader/s1_reader.py:1258 in load_single_burst                  │
│                                                                                                  │
│   1255 │   if not os.path.exists(orbit_path):                                                    │
│   1256 │   │   raise FileNotFoundError(f'{orbit_path} not found')                                │
│   1257 │                                                                                         │
│ ❱ 1258 │   burst = burst_from_combined_xml(data_path, metadata_path, orbit_path, flag_apply_eap  │
│   1259 │   return burst                                                                          │
│   1260                                                                                           │
│   1261 def load_bursts(path: str, orbit_path: str, swath_num: int, pol: str = 'vv',              │
│                                                                                                  │
│ /home/scott/repos/s1-reader/src/s1reader/s1_reader.py:914 in burst_from_combined_xml             │
│                                                                                                  │
│    911 │   # Load RFI information if available                                                   │
│    912 │   tree_rads = get_subxml_from_burst_metadata(tree_metadata, 'product', subswath, pol)[  │
│    913 │   if tree_rads is not None:                                                             │
│ ❱  914 │   │   annotation_datasets['burst_rfi_info_swath'] = SwathRfiInfo.from_et(tree_rads,     │
│    915 │   │   │   │   │   │   │   │   │   │   │   │   │   tree_lads,                            │
│    916 │   │   │   │   │   │   │   │   │   │   │   │   │   ipf_version)                          │
│    917 │   else:                                                                                 │
│                                                                                                  │
│ /home/scott/repos/s1-reader/src/s1reader/s1_annotation.py:647 in from_et                         │
│                                                                                                  │
│    644 │   │                                                                                     │
│    645 │   │   header_rfi = et_rfi.find('rfiBurstReportList')                                    │
│    646 │   │   if header_rfi is None:                                                            │
│ ❱  647 │   │   │   raise ValueError('Cannot locate `rfiBurstReportList` '                        │
│    648 │   │   │   │   │   │   │    'in the RFI annotation')                                     │
│    649 │   │                                                                                     │
│    650 │   │   # Start to load RFI information                                                   │
╰───────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Cannot locate `rfiBurstReportList` in the RFI annotation

other initial thoughts-

• lets export the load_single_burst in __init__.py so we can use it just by doing import s1reader
• if possible, it could be nice to separate this into an asf.py module. You seemed to have nicely fit stuff in here, but it's been talked about how s1_reader is already pretty bloated now. Maybe we can push this off til a larger refactor happens.
• not sure if you thought of any nice ways to get the orbit yet. Not a big deal now (the orbit downloading within s1reader is broken at the moment anyway), but I got my file by just copying the datetime string in the burst filename and doing

$ eof --date 20230922T015859 --mission S1A

[forrestfwilliams]

Hey @scottstanie I'm back from vacation now. Thanks for the suggestion and finding the bug. I've done the following:

1. Fixed the bug you found (it was a copy/paste error)
2. Made load_single_burst a top-level import
3. Created a script for generating test data. I've attached both the script and a sample product.

prep_data.py.txt S1_292315_IW3_20230922T015859_VV_9961-BURST.zip

BTW, the prep_data script shows how you can use hyp3lib (or your own version of the hyp3lib function) to download orbit data from ESA's new service.

[forrestfwilliams]

I'm happy to move the new functionality into an asf.py file if you think it keeps the repo cleaner. Just let me know what you'd prefer!

[scottstanie]

In [12]: import s1reader.s1_reader
In [13]: s1reader.s1_reader.load_single_burst("S1_292315_IW3_20230922T015859_VV_9961-BURST.tiff", "S1_292315_IW3_20230922T015859_VV_9961-BURST.xml", "S1A_OPER_AUX_POEORB_OPOD_20231012T080638_V20230921T225942_20230923T005942.EOF")
Out[13]: Sentinel1BurstSlc(burst_id=t137_292315_iw3)

looks like it's working for me :+1: haven't tried to run any geocoding with it or anything though. S1A_OPER_AUX_POEORB_OPOD_20231012T080638_V20230921T225942_20230923T005942.EOF.zip S1_292315_IW3_20230922T015859_VV_9961-BURST.xml.zip S1_292315_IW3_20230922T015859_VV_9961-BURST.tiff.zip

To help make a test for this, I zipped up my test data, and i zeroed out the .tiff so it's only 10 kb. Posting here for now

[forrestfwilliams]

Great, glad to hear it's working for you! Anything I can help with before the merge?

[scottstanie]

Nope just need another :+1: from someone to unblock the button

[forrestfwilliams]

Closing this PR as this work is superseded by the creation of burst2safe. You can use burst2safe to create custom SAFE files that are compatible with s1-reader. For example:

burst2safe S1_136231_IW2_20200604T022312_VV_7C85-BURST --all-anns

Importantly the --all-anns must be used to ensure s1-reader compatibility.