MiaplPy
MiaplPy copied to clipboard
Published
20 hours ago •
insarlab
insarlab
out of data range error although subset is within SLC area
I am getting an out of data range error, although I have selected a subset that is fully within the SLCs. It turns out that the LENGTH and WIDTH in the *.rsc files are too small. Some multilooking seems to have been applied and this is what causes the problem? I darkly remember that we had this before, but I don't remember details. Any idea?
miaplpyApp.py /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/samples/unittestGalapagosSenDT128.template --dir miaplpy --start load_data
--------------------------------------------------
_________________________________________________
/## /## /## / ## /#######
| ### /###|__/| | ##| ##__ ##
| #### /#### /##|/ ##### /## | ##| ## \ ## /## /##
| ## ##/## ##| ##| ##__ ### /####### | ##| #######/| ## | ##
| ## ###| ##| ##| ##__ ###| ##__ ##| ##| ## | ## | ##
| ##\ # | ##| ##| ## \###| ## \##| ##| ## | ## | ##
| ## \/ | ##| ##| ##### ##| #######/| ##| ## | #######
|__/ |__/|__/| \____/|_/| ##____/ |__/|__/ \____ ##
| ## /## | ##
| ## | ######/
|__/ \______/
Miami Non-Linear Phase Linking software in Python
MiaplPy 0.2.0, 2021-09-14
_________________________________________________
--RUN-at-2022-10-14 21:04:15.426400--
Current directory: /scratch/05861/tg851601/unittestGalapagosSenDT128
Run routine processing with arg_parser.py on steps: ['load_data', 'phase_linking', 'concatenate_patches', 'generate_ifgram', 'unwrap_ifgram', 'load_ifgram', 'ifgram_correction', 'invert_network', 'timeseries_correction']
Remaining steps: ['phase_linking', 'concatenate_patches', 'generate_ifgram', 'unwrap_ifgram', 'load_ifgram', 'ifgram_correction', 'invert_network', 'timeseries_correction']
Project name: unittestGalapagosSenDT128
--------------------------------------------------
20221014:210415 * miaplpyApp.py /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/samples/unittestGalapagosSenDT128.template --dir miaplpy --start load_data
Project name: unittestGalapagosSenDT128
Go to work directory: /scratch/05861/tg851601/unittestGalapagosSenDT128/miaplpy
copy default template file /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/defaults/smallbaselineApp.cfg to work directory
read custom template file: /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/samples/unittestGalapagosSenDT128.template
update default template based on input custom template
mintpy.compute.cluster: auto --> local
mintpy.compute.numWorker: auto --> 6
mintpy.load.processor: auto --> isce
mintpy.load.autoPath: auto --> yes
mintpy.subset.lalo: auto --> -0.86:-0.81,-91.19:-91.13
mintpy.reference.lalo: auto --> -0.82,-91.14
mintpy.troposphericDelay.method: auto --> no
mintpy.save.kmz: auto --> yes
mintpy.save.hdfEos5: auto --> yes
mintpy.save.hdfEos5.update: auto --> yes
copy unittestGalapagosSenDT128.template to inputs directory for backup.
copy smallbaselineApp.cfg to inputs directory for backup.
copy unittestGalapagosSenDT128.template to pic directory for backup.
copy smallbaselineApp.cfg to pic directory for backup.
read default template file: /scratch/05861/tg851601/unittestGalapagosSenDT128/miaplpy/smallbaselineApp.cfg
copy default template file /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/objects/../defaults/miaplpyApp.cfg to work directory
update default template based on input custom template
miaplpy.load.processor: auto --> isce
miaplpy.load.autoPath: auto --> yes
No new option value found, skip updating /scratch/05861/tg851601/unittestGalapagosSenDT128/miaplpy/custom_smallbaselineApp.cfg
copy miaplpyApp.cfg to inputs directory for backup.
read default template file: /scratch/05861/tg851601/unittestGalapagosSenDT128/miaplpy/miaplpyApp.cfg
miaplpy.load.compression: auto --> no
miaplpy.load.autoPath: yes --> yes
miaplpy.subset.lalo: auto --> -0.86:-0.81,-91.19:-91.13
SAR platform/sensor : Sen
processor: isce
check auto path setting for Univ of Miami users for processor: isce
--------------------------------------------------
prepare metadata files for isce products
prep_slc_isce.py -s /scratch/05861/tg851601/unittestGalapagosSenDT128/merged/SLC -f *.slc.full -m /scratch/05861/tg851601/unittestGalapagosSenDT128/reference/IW1.xml -b /scratch/05861/tg851601/unittestGalapagosSenDT128/baselines -g /scratch/05861/tg851601/unittestGalapagosSenDT128/merged/geom_reference
['/scratch/05861/tg851601/unittestGalapagosSenDT128/reference/data.rsc'] exists and is newer than ['/scratch/05861/tg851601/unittestGalapagosSenDT128/reference/IW1.xml'] --> skip.
prepare .rsc file for geometry files
read perp baseline time-series from /scratch/05861/tg851601/unittestGalapagosSenDT128/baselines
preparing RSC file for *.slc.full
[==================================================] 20160605_20160828 0s / 0s
Done.
Traceback (most recent call last):
File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/miaplpyApp.py", line 801, in <module>
main()
File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/miaplpyApp.py", line 82, in main
app.open()
File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/miaplpyApp.py", line 176, in open
self.date_list, self.num_pixels, self.metadata = read_initial_info(self.workDir, self.templateFile)
File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/objects/utils.py", line 1223, in read_initial_info
metadata = read_subset_box(iDict)
File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/objects/utils.py", line 1156, in read_subset_box
pix_box = coord.check_box_within_data_coverage(pix_box)
File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/objects/coord.py", line 477, in check_box_within_data_coverage
raise ValueError(msg)
ValueError: ERROR: input index is out of data range!
data range in (x0,y0,x1,y1): (0, 0, 1364, 565)
subset range in (x0,y0,x1,y1): (7009, 1486, 8912, 2010)
data range in (W, N, E, S): None
subset range in (W, N, E, S): None
unittestGalapagosSenDT128/merged/SLC/20160605[1121] cat *.full.rsc | grep -E 'LENGTH|length'
FILE_LENGTH 565
LENGTH 565
length 2826
For comparison, here for a different dataset which works fine:
MiamiSenAT48/merged/SLC/20150921[1029] cat *.full.rsc | grep -E 'LENGTH|length'
FILE_LENGTH 5531
LENGTH 5531
length 5531
Here the *template file for the problem case:
cat /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/samples/unittestGalapagosSenDT128.template
######################################################
cleanopt = 0 # [ 0 / 1 / 2 / 3 / 4] 0,1: none 2: keep merged,geom_master,SLC 3: keep MINTPY 4: everything
processor = isce
ssaraopt.platform = SENTINEL-1A,SENTINEL-1B
ssaraopt.relativeOrbit = 128
ssaraopt.startDate = 20160601
ssaraopt.endDate = 20160831
insarmaps_flag = False
image_products_flag = False
######################################################
topsStack.boundingBox = -1 -0.6 -91.9 -90.7 # -1 0.15 -91.9 -90.6
topsStack.boundingBox = -0.81 -0.80 -90.9 -90.86 # -1 0.15 -91.9 -90.6
topsStack.subswath = 1 # '1 2'
topsStack.numConnections = 3 # comment
topsStack.azimuthLooks = 5 # comment
topsStack.rangeLooks = 15 # comment
topsStack.filtStrength = 0.2 # comment
topsStack.unwMethod = snaphu # comment
topsStack.coregistration = auto # [NESD geometry], auto for NESD
topsStack.slcDir = $TESTDATA_ISCE/unittestGalapagosSenDT128/SLC
######################################################
mintpy.load.autoPath = yes
mintpy.compute.cluster = local #[local / slurm / pbs / lsf / none], auto for none, cluster type
mintpy.compute.numWorker = 6 #[int > 1 / all], auto for 4 (local) or 40 (non-local), num of workers
mintpy.reference.lalo = -0.82,-91.14 # S of SN
mintpy.networkInversion.parallel = yes #[yes / no], auto for no, parallel processing using dask
mintpy.troposphericDelay.method = no # pyaps #[pyaps / height_correlation / base_trop_cor / no], auto for pyaps
mintpy.save.hdfEos5 = yes #[yes / update / no], auto for no, save timeseries to UNAVCO InSAR Archive format
mintpy.save.hdfEos5.update = yes #[yes / no], auto for no, put XXXXXXXX as endDate in output filename
mintpy.save.hdfEos5.subset = yes #[yes / no], auto for no, put XXXXXXXX as endDate in output filename
mintpy.save.kmz = yes #[yes / no], auto for yes, save geocoded velocity to Google Earth KMZ file
######################################################
miaplpy.load.processor = isce
miaplpy.load.autoPath = yes
miaplpy.compute.num_workers = 46
miaplpy.subset.lalo = -0.86:-0.81,-91.19:-91.13 #[31.5:32.5,130.5:131.0 / no], auto for no
miaplpy.interferograms.networkType = sequential # [single_reference, sequential, combine, list] default: single_reference
@falkamelung I don't understand why you are comparing data length for unittestGalapagosSenDT128 and MiamiSenAT48. They are definitely different
I run unittestGalapagosSenDT128 data without problem and did not get this error. You might not be using the last version of MiaplPy or your DEM is not downloaded properly:
unittestGalapagosSenDT128/merged/SLC/20160605[1070] cat *.full.rsc | grep -E 'LENGTH|length'
FILE_LENGTH 2826
LENGTH 2826
length 2826
Also these 2 options do not have overlap:
topsStack.boundingBox = -0.81 -0.80 -90.9 -90.86
miaplpy.subset.lalo = -0.86:-0.81,-91.19:-91.13
however it is not a big problem for this dataset because isce uses a larger subset than the bbox in template and still covers the subset defined for miaplpy
Hi @mirzaees Thank you for running this. I am glad this it is still working for you! Are you using the committed unittestGalapagosSenDT128.template? If not, can you please post yours?
I ran with a new DEM and I am still getting the same error and have multi looked LENGTH.
/unittestGalapagosSenDT128/merged/SLC/20160605[1097]
grep -E 'LENGTH|length' *rsc
FILE_LENGTH 565
LENGTH 565
length 2826
I did check for latest versions. So maybe my installation got screwed and I should do a new one.
Hi @mirzaees I found the problem. The issue occurs if you run MintPy prior to running MiaplPy.
The Mintpy prep_isce* module creates an /reference/data.rsc. If that exists then MiaplPy has a problem. I did not look at the code but it looks that the MiaplPy prep_isce* module does not create it if it exists, or similar, which looks like an easy fix.
The offending attribute seems to be WIDTH which is created by the Mintpy module but not by the MiaplPy module.
unittestGalapagosSenDT128
grep WIDTH reference/data.rsc mintpy_data.rsc
mintpy_data.rsc:WIDTH 1364
If both MiaplPy and Mintpy always overwrite all *rsc files there should not be an issue.