Problems importing GMTSAR data

[WangJunyi2022]

Hello， I processed the image using GMTSAR, and after processing, converted the los.grd deformation to m, and generated the los.enu data. But after I import the data, the following problem occurs.

(grond) wang@K:/media/wang/dec2022/merge$ spool --load unwrap_ll.grd
Traceback (most recent call last):
  File "/home/wang/anaconda3/envs/grond/bin/spool", line 8, in <module>
    sys.exit(main())
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/spool/__main__.py", line 95, in main
    spool(import_file=ns.load)
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/spool/spool.py", line 405, in spool
    spool_app = Spool(*args, **kwargs)
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/spool/spool.py", line 48, in __init__
    self.importScene(import_file)
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/spool/spool.py", line 64, in importScene
    self.spool_win.model.importFile(filename)
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/spool/scene_model.py", line 174, in importFile
    self.setScene(Scene.import_data(filename))
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/scene.py", line 494, in dynclassmethod
    return func(Scene(), *args, **kwargs)
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/scene.py", line 1013, in _import_data
    data = module.read(path, **kwargs)
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/kite/scene_io.py", line 815, in read
    grd = netcdf.netcdf_file(self._getDisplacementFile(path),
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/scipy/io/_netcdf.py", line 277, in __init__
    self._read()
  File "/home/wang/anaconda3/envs/grond/lib/python3.8/site-packages/scipy/io/_netcdf.py", line 598, in _read
    raise TypeError("Error: %s is not a valid NetCDF 3 file" %
TypeError: Error: /media/wang/WJY/menyuandec2022/merge/unwrap_ll.grd is not a valid NetCDF 3 file

How can I import the data?

[hvasbath]

Hi @WangJunyi2022!

Thank you for your message! The problem is that likely the newer GMTSAR version stores the data in netcdf4 right? Can you please check? If that is the case then Kite would need to be extended to be able to read netcdf4, as the scipy netcdf reader (used in kite) does not support netcdf4. That would need to be added to scene_io.py line 815. I quick workaround would be if you could somehow tell GMTSAR to store the data in netcdf3 format.

Regards! Hannes

[wasjabloch]

xarray handles NetCDF4 super well. It is very powerful and mature.

[WangJunyi2022]

Thanks! It is the version of InSAR that I use, and there is no problem with GMT5SAR.

[hvasbath]

InSAR has no version. The problem is that GMTSAR stores the data in netcdf4 format, which is currently not supported by KITE. So either you need to find out if GMTSAR can also store data in netcdf3 or you/someone need to work on a patch for KITE. As we, the maintainers of KITE are up to the brim with work- we wont have time to do that for you in the near time.

[WangJunyi2022]

Sorry, I spelt it wrong. I used GMTSAR version 6.1. I would like to say that due to the software version, the gmtsar6 default generated file is netcdF4 and cannot be opened with kite. But there should be no problem using the kite to turn on the GMTSAR5 for data processing. In addition, when installing gmtsar5, I could not install it on ubuntu20 due to version migration, but I installed gmtsar5.8 on ubuntu16. (I hope my mistake will save others some trouble.) Good luck !

[Meysam-Amiri]

I have a solution for importing GMTSAR data into Kite. Here's a step-by-step guide on how to do it:

1. Copy the following files from the merge directory into a separate directory:

   • los_ll.grd
   • supermaster.PRM
   • S1_20211109_ALL_F3.LED
   • dem.grd

2. Create a new file named "gmt.conf" and add the following lines to it:

#
# GMT 6.3.0 Defaults file
#
# COLOR Parameters
#
COLOR_MODEL                    = hsv
#
# FORMAT Parameters
#
FORMAT_GEO_OUT                 = D
#
# I/O Parameters
#
IO_NC4_CHUNK_SIZE              = classic
#
# PostScript Parameters
#
PS_MEDIA                       = a2

Note that you can Replace the "IO_NC4_CHUNK_SIZE" value in the existing "gmt.conf" file from the merge directory with the line:

IO_NC4_CHUNK_SIZE              = classic

By setting "IO_NC4_CHUNK_SIZE" to "classic," the grd files will be saved in Netcdf3 format.

4. Convert the "los_ll.grd" file from millimeters (mm) to meters (m):

gmt grdmath los_ll.grd 0.001 MUL = los_ll_m.grd

5. Extract the lon, lat, and elevation data from the "dem.grd" file and apply it to the converted "los_ll_m.grd" file:

gmt grd2xyz los_ll_m.grd | gmt grdtrack -Gdem.grd | awk '{print $1, $2, $4}' | SAT_look supermaster.PRM -bos > 20211109.los.enu

6. Save the "los_ll_m.grd" as a file np).

spool --load los_ll_m.grd
save los_ll_m.npz

After following these steps, you should have successfully imported the GMTSAR data into Kite, and the results will be saved in Netcdf3 format and a npz file. If you would like to cut grd file for a small region: gmt grdcut los_ll_m.grd -Rmin_lon/max_lon/min_lat/max_lat -Glos_ll_m_cut.grd

[ditafaith]

gmt grd2xyz los_ll_m.grd | gmt grdtrack -Gdem.grd | awk '{print $1, $2, $4}' | SAT_look supermaster.PRM -bos > 20211109.los.enu

Hi @Meysam-Amiri , I am using ALOS data, is there any difference command ? what is SAT_look ? and what is the purpose to create gmt.conf file ?

[Meysam-Amiri]

gmt grd2xyz los_ll_m.grd | gmt grdtrack -Gdem.grd | awk '{print $1, $2, $4}' | SAT_look supermaster.PRM -bos > 20211109.los.enu

Hi @Meysam-Amiri , I am using ALOS data, is there any difference command ? what is SAT_look ? and what is the purpose to create gmt.conf file ?

Hi @ditafaith. I am not sure about ALOS data. But, I think the procedure would be the same. SAT_look is one of the GMTSAR commands (https://github.com/gmtsar/gmtsar/blob/master/gmtsar/SAT_look.c). The purpose of creating gmt.conf with new adding IO_NC4_CHUNK_SIZE = classic is to create required grd files in Netcdf 3 format which is readable by kite.

[ditafaith]

@Meysam-Amiri I got the point, yes..it is running. However there is no image appears. It looks similar when I tried to run example from kite's website.

[Meysam-Amiri]

How did you import data to kite? Could you provide some information about the commands and interferogram that you are trying to import to kite?

[ditafaith]

How did you import data to kite? Could you provide some information about the commands and interferogram that you are trying to import to kite?

What do you mean import to kite ? I just follow the guidance, directly open file from gmtsar. https://pyrocko.org/kite/docs/current/quickstart.html

[Meysam-Amiri]

How did you import data to kite? Could you provide some information about the commands and interferogram that you are trying to import to kite?

What do you mean import to kite ? I just follow the guidance, directly open file from gmtsar. https://pyrocko.org/kite/docs/current/quickstart.html

You must prepare some files before using spool --load command. Please read where it explained GMTSAR: https://pyrocko.org/kite/docs/current/reference/kite.scene.html#module-kite.scene_io GMTSAR

Reading GMTSAR grid files.

Note Expects: Displacement grid (NetCDF, *los_ll.grd) in meter (in case use “gmt grdmath los_cm_ll.grd 0.01 MUL = los_m_ll.grd’)

LOS binary data (see instruction, *.los.enu)

Calculate the corresponding unit look vectors with GMT5SAR SAT_look:

gmt grd2xyz los_ll.grd | gmt grdtrack -Gdem.grd |
awk {'print $1, $2, $4'} |
SAT_look 20050731.PRM -bos > 20050731.los.enu

[ditafaith]

How did you import data to kite? Could you provide some information about the commands and interferogram that you are trying to import to kite?

What do you mean import to kite ? I just follow the guidance, directly open file from gmtsar. https://pyrocko.org/kite/docs/current/quickstart.html

You must prepare some files before using spool --load command. Please read where it explained GMTSAR: https://pyrocko.org/kite/docs/current/reference/kite.scene.html#module-kite.scene_io GMTSAR

Reading GMTSAR grid files.

Note Expects: Displacement grid (NetCDF, *los_ll.grd) in meter (in case use “gmt grdmath los_cm_ll.grd 0.01 MUL = los_m_ll.grd’)

LOS binary data (see instruction, *.los.enu)

Calculate the corresponding unit look vectors with GMT5SAR SAT_look:

gmt grd2xyz los_ll.grd | gmt grdtrack -Gdem.grd | awk {'print $1, $2, $4'} | SAT_look 20050731.PRM -bos > 20050731.los.enu

Yes, I did. This is similar as you explained in this threads as well. I changed to netcdf3 for los_ll files. However, I can't read *.los.enu file due to binary file. Is it correct ? There is no different size between original los file and the new los in netcdf3 version.

[Meysam-Amiri]

What is the output of gmt grdinfo los_ll*.grd? No, it does not change the size.

[ditafaith]

I'd be glad to getting insighful comments from @Meysam-Amiri and others regarding this case.

[Meysam-Amiri]

I can check it if you share your results with me. Here is my email address : @.***

On Sun, Aug 6, 2023, 17:30 ditafaith @.***> wrote:

I'd be glad to getting insighful comments from @Meysam-Amiri https://github.com/Meysam-Amiri and other regarding this case.

— Reply to this email directly, view it on GitHub https://github.com/pyrocko/kite/issues/73#issuecomment-1666870027, or unsubscribe https://github.com/notifications/unsubscribe-auth/AY6U2KYADEKBHXEVLEYPPQ3XT6PQBANCNFSM5S4DMRGQ . You are receiving this because you were mentioned.Message ID: @.***>

[ditafaith]

I can check it if you share your results with me. Here is my email address : @.***

On Sun, Aug 6, 2023, 17:30 ditafaith @.***> wrote:

I'd be glad to getting insighful comments from @Meysam-Amiri https://github.com/Meysam-Amiri and other regarding this case.

— Reply to this email directly, view it on GitHub https://github.com/pyrocko/kite/issues/73#issuecomment-1666870027, or unsubscribe https://github.com/notifications/unsubscribe-auth/AY6U2KYADEKBHXEVLEYPPQ3XT6PQBANCNFSM5S4DMRGQ . You are receiving this because you were mentioned.Message ID: @.***>

You can't post email address through github, since they automatically changed *. Do you mean result netcdf3 from GMTSAR ?

[Meysam-Amiri]

[email protected] Yes please send me grd file. Maybe I can ssh and produce grd files. Anyway, Let's be in contact and solve it through email.

[Meysam-Amiri]

You can also look at https://hive.pyrocko.org/pyrocko-support/channels/kite and ask there for help. Please DM me there to chat easily.