Issue stackSentinel -> run 12

[alevasca]

Hi all,

When running run_12_merge_reference_secondary_slc I got this:

Traceback (most recent call last):
  File "/sw/jessie-x64/isce2-040321/contrib/stack/topsStack/SentinelWrapper.py", line 251, in <module>
    main(args.start,args.end)
  File "/sw/jessie-x64/isce2-040321/contrib/stack/topsStack/SentinelWrapper.py", line 242, in main
    cfgParser.runCmd()
  File "/sw/jessie-x64/isce2-040321/contrib/stack/topsStack/SentinelWrapper.py", line 54, in runCmd
    func_modules.main(self.funcParams[section])
  File "/opt/cen/sw/sw/jessie-x64/isce2-040321/contrib/stack/topsStack/mergeBursts.py", line 369, in main
    inps.reference, ifg.numberOfBursts, reference.numberOfBursts))
ValueError: /scratch/uni/Sentinel1/stackSentinel/KJ/2015/DESC/coreg_secondarys/20150402 has different number of bursts (3) than the reference (5)

I'm working with ISCE2 v.2.5. I did not get this issue with previous versions. My area of interest is in the middle of the second swath and there is no shift in azimuth direction (I mean, all SLC images correspond to the same frame). I see that he SLC files with the smallest size (e.g. 1.97GB, 2.08 GB, ...) are linked with this error. What could I do to fix this issue?

Thanks in advance for any help!

[rtburns-jpl]

I wonder if this is due to https://github.com/isce-framework/isce2/commit/55b9d2c03795429dc64f7a69883a4a7f413a0fdf. If you're able to compile isce2, could you maybe try processing before and after that commit to see if that is the cause?

[hardreddata]

I also had this issue. Some of the early captures in my areas of interest in 2015 & 2016 have 10 bursts and/or the scene is shifted along the relative orbit path.

To explore this I patched the ISCE 2.5.1 conda package, rolling back extractCommonValidRegion.py and mergeBursts.py it looks like it is working though the unwrap step is still running. Certainly I am seeing unwrapped data for dates previously discarded.

[bjmarfito]

See this post in ISCE forums providing explanation to the warning given to you (http://earthdef.caltech.edu/boards/4/topics/3706?r=3709#message-3709).

[hardreddata]

Thanks @bjmarfito

I am not sure the actual behaviour agrees with that forum post. I have the two consecutive SLCs in my source folder and, pushing through the warning, the merged folder would only populate from December 2016 onwards.

When I get a chance I will try and find a concise example.

[bjmarfito]

I would like to see an example @RussellGrew.

For now, I recommend you to have a larger bounding box when downloading the Sentinel-1 SLC images than the bounding box of your study area in stackSentinel.py. This might solve your problem.

[hardreddata]

Hi @bjmarfito

This came together quicker than I had anticipated. Thanks for your interest.

I made a clean conda environment, did the install and then pip install shapely per https://github.com/isce-framework/isce2/issues/275 (I can move that to the conda-forge issue tracker if you like)

conda list | grep isce2
isce2                     2.5.2            py39hee44cce_0    conda-forge

I have these SLC from relative orbit 147 over Sydney, Australia. I put them in their own test folder.

S1A_IW_SLC__1SDV_20160922T191509_20160922T191536_013169_014EF0_C52B.zip
S1A_IW_SLC__1SDV_20160922T191533_20160922T191600_013169_014EF0_D99C.zip
S1A_IW_SLC__1SDV_20161004T191506_20161004T191533_013344_01547C_6BA6.zip
S1A_IW_SLC__1SDV_20161004T191531_20161004T191559_013344_01547C_BE47.zip
S1A_IW_SLC__1SDV_20161016T191506_20161016T191533_013519_015A07_98E6.zip
S1A_IW_SLC__1SDV_20161016T191531_20161016T191559_013519_015A07_0C3C.zip

On reflection one assumption I have made is that I don't need to go into another relative orbit.

I prepare the stack in a new folder.

dem.py -a stitch -b -35 -33 149 152 -r -s 1 --correct -k

stackSentinel.py -s /mnt/e/Sentinel-1A/test -o /mnt/e/Sentinel-1A/orbit -a /mnt/e/Sentinel-1A/auxl -d demLat_S35_S33_Lon_E149_E152.dem.wgs84  -b '-33.85 -33.80 151.05 151.10' 

I understand SAFE_files.txt shows that we only need the two SLC for the first date.

/mnt/e/Sentinel-1A/test/S1A_IW_SLC__1SDV_20160922T191509_20160922T191536_013169_014EF0_C52B.zip
/mnt/e/Sentinel-1A/test/S1A_IW_SLC__1SDV_20160922T191533_20160922T191600_013169_014EF0_D99C.zip
/mnt/e/Sentinel-1A/test/S1A_IW_SLC__1SDV_20161004T191531_20161004T191559_013344_01547C_BE47.zip
/mnt/e/Sentinel-1A/test/S1A_IW_SLC__1SDV_20161016T191531_20161016T191559_013519_015A07_0C3C.zip

Run 11 shows the warning.

sh run_11_extract_stack_valid_region
...
creating  /mnt/h/stack_test/stack
checking the number of bursts in coreg_secondarys against the one in reference
WARNING: 20161004 has different number of bursts (2) than the reference reference (3) for swath 1 --> exclude it for common region calculation
WARNING: 20161016 has different number of bursts (2) than the reference reference (3) for swath 1 --> exclude it for common region calculation
******************
swath:  1
writing  /mnt/h/stack_test/stack/IW1.xml

Run 12 fails.

sh run_12_merge_reference_secondary_slc
...
ValueError: /mnt/h/stack_test/coreg_secondarys/20161016 has different number of bursts (2) than the reference (3)

and then the merged/SLC folder only contains data for 20160922. Run 14 also complains when trying to merge the interferograms. The coherence filtering and unwrap steps work but only on the interferograms which exist.

I have explored forcing reference date with -m and also somehow tricking it by choosing coordinates away from boundaries after study of https://github.com/isce-framework/isce2/issues/152 but nothing worked consistently.

The orbit files are recently downloaded and are all S1A_OPER_AUX_POEORB_OPOD.

I hope this helps! Please let me know if I can somehow assist.

[hfattahi]

Trying to copy my response to similar questions elsewhere. It may help here as well:

The message is clear, the number of bursts in the coregistered secondary and the reference image are not the same. If we allow this go on then you would end up with a smaller area after merging the stack. Imagine you have 100 dates and all of them have 20 bursts except for one date which has 10 bursts. If we allow to proceed without this check, then we would end up with 10 bursts for all 100 dates. But by eliminating that 1 date we will have 99 dates with 20 bursts. Of course this is an extreme example.

Note that the Sentinel-1 frames are not fixed and they may move from date to date. Basically the same frame number does may not have the same bursts on ground in different dates. That is the issue with Sentinel-1 and the reason that the processing has been complicated.

Because of the issue with Sentinel-1 frames, the rule of thumb for processing Sentinel-1 data is always download data more than what you want. Don’t rely on frame number at all. Use bounding box to download the data and always for downloading data use a bounding box larger than what you will use for processing. For example your download bounding box can be 0.25-0.5 degrees on both sound and north for downloading data. This way you can make sure that there is enough bursts to cover your area.

[bjmarfito]

Thank you @hfattahi for providing a thorough explanation regarding this issue.

[hardreddata]

Thanks @hfattahi

Changing the bounding box in the above example to -b '-33.9 -33.7 151.0 151.2' solves the issue over those three dates.

Given your knowledge of the common area algorithm, for this data set, will zooming out more than this always work? Naively I wonder if the offset between the bursts could still be an issue for some chosen coordinates?

I am experimenting with zooming out more to see if this holds true.

I see '-34 -33 150.0 151.2' and '-34 -33.6 150.8 151.2' are both good.