Add additional filter to support Landsat 7 SLC-off imagery

[jhkennedy]

fixes #52

[alex-s-gardner]

@jhkennedy would you be able to test of these 2 images [band 8]? I1 = LE07_L2SP_062018_20210717_20210812_02_T1 I2 = LE07_L2SP_062018_20210701_20210728_02_T1

[leiyangleon]

Looks good to me.

[jhkennedy]

Since we're processing Level-1 now, the example pair would be: LE07_L1TP_062018_20210717_20210812_02_T1 LE07_L1TP_062018_20210701_20210728_02_T1

[jhkennedy]

In order to better test Landsat-7, we can compare directly to version 1 ITS_LIVE products:

• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_061018_20040321_20160927_01_T1_X_LE07_L1TP_062018_20040225_20160927_01_T1_G0240V01_P076.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_063018_20040911_20160926_01_T1_X_LE07_L1TP_063018_20040810_20160926_01_T1_G0240V01_P075.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_061018_20050324_20160925_01_T1_X_LE07_L1TP_061018_20040321_20160927_01_T1_G0240V01_P072.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_062018_20040803_20160926_01_T1_X_LE07_L1TP_061018_20040625_20160926_01_T1_G0240V01_P068.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_063018_20040319_20160926_01_T1_X_LE07_L1TP_062018_20040225_20160927_01_T1_G0240V01_P067.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_061018_20040625_20160926_01_T1_X_LE07_L1TP_061018_20030810_20160927_01_T1_G0240V01_P066.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_061018_20040711_20160926_01_T1_X_LE07_L1TP_061018_20040625_20160926_01_T1_G0240V01_P066.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_063018_20040911_20160926_01_T1_X_LE07_L1TP_062018_20040803_20160926_01_T1_G0240V01_P062.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_062018_20040803_20160926_01_T1_X_LE07_L1TP_063018_20040623_20160926_01_T1_G0240V01_P062.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_062018_20040803_20160926_01_T1_X_LE07_L1TP_061018_20040711_20160926_01_T1_G0240V01_P061.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_061018_20050511_20160925_01_T1_X_LE07_L1TP_061018_20040625_20160926_01_T1_G0240V01_P054.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_061018_20040711_20160926_01_T1_X_LE07_L1TP_061018_20030810_20160927_01_T1_G0240V01_P054.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_061018_20040625_20160926_01_T1_X_LE07_L1TP_061018_20040321_20160927_01_T1_G0240V01_P053.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_061018_20050425_20160925_01_T1_X_LE07_L1TP_061018_20040321_20160927_01_T1_G0240V01_P053.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_062018_20050416_20160925_01_T1_X_LE07_L1TP_062018_20040803_20160926_01_T1_G0240V01_P052.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_063018_20050509_20160924_01_T1_X_LE07_L1TP_063018_20040810_20160926_01_T1_G0240V01_P051.nc
• http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/velocity_image_pair/landsat/v00.0/32607/LE07_L1TP_062018_20040803_20160926_01_T1_X_LE07_L1TP_062018_20040718_20160926_01_T1_G0240V01_P051.nc

I've also attached a CSV of these scene name. Importantly, these are Collection 1 scene names, so we'll need to convert them to collection 2 names (I think just processing date has changed) to run them.

L7_testpairs.csv

[jhkennedy]

Here's the Landsat-7 naming convention, which holds across collections: https://www.usgs.gov/faqs/what-naming-convention-landsat-collections-level-1-scenes

So it looks like just the collection number and possibly the processing date needs to change. You can use this function to confirm you've got collection 2 scene names: https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/process.py#L51

[jhkennedy]

Alright, I believe L7 processing using the Wallis-no-data-fill preprocessor is working as intended at this point. Here is an example product with outputs, including the source scene and the associated ITS_LIVE V1 data: https://drive.google.com/file/d/1FTtd8kS5Tz2OLtAzL1P2uXiH2pWsv0Fx/view?usp=sharing

And the browse image for this example:

[alex-s-gardner]

@jhkennedy this looks really good. The striping in V02 is due to a less aggressive interpolation for a 120m grid. The loss of data over land in the right of the image is a bit concerning but difficult to trace. The V02 input images have been reprocessed since v01... that might explain some of the difference. I also noticed the the image order in the file name is swapped. I vaguely remember discussing this previously by have now forgotten what we decided on that.

[jhkennedy]

@alex-s-gardner In HyP3-autoRIFT we enforce "earliest scene first" as the reference image, and we build the file name with reference x secondary.