isce3 icon indicating copy to clipboard operation
isce3 copied to clipboard

Published 20 hours ago • verified publisher icon isce-framework

alos2_to_nisar_l1.py can't convert alos2 ceos to h5

Open zyxgis opened this issue 2 years ago • 4 comments

os : centos 7 python: 3.9 cxx: gcc11

i use alos2_to_nisar_l1.py to convert alos2 ceos to h5, but i get the follow error:

$ /python/bin/python3 /home/testUser/isce/share/nisar/examples/alos2_to_nisar_l1.py -l HH -i /home/testUser/test/ceos/0000471428_001001_ALOS2441290770-220725/ -o /home/testUser/test/ceos/output-hh2.h5 looking for available polarizations... found polarization: HH HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0: #000: H5O.c line 1175 in H5Ovisit_by_name3(): invalid location identifier major: Invalid arguments to routine minor: Inappropriate type #001: H5VLint.c line 1741 in H5VL_vol_object(): invalid identifier major: Invalid arguments to routine minor: Inappropriate type HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0: #000: H5D.c line 132 in H5Dcreate2(): invalid location identifier major: Invalid arguments to routine minor: Inappropriate type Traceback (most recent call last): File "/home/testUser/isce/share/nisar/examples/alos2_to_nisar_l1.py", line 720, in process(args=args) File "/home/testUser/isce/share/nisar/examples/alos2_to_nisar_l1.py", line 139, in process orbit = construct_nisar_hdf5(args.outh5, leader) File "/home/testUser/isce/share/nisar/examples/alos2_to_nisar_l1.py", line 253, in construct_nisar_hdf5 set_h5_orbit(orbit_group, orbit) File "/home/testUser/isce/share/nisar/examples/alos_to_nisar_l0b.py", line 141, in set_h5_orbit orbit.save_to_h5(group) RuntimeError: Caught an unknown exception!

zyxgis avatar May 19 '23 06:05 zyxgis

My guess is that your h5py and isce3 are linked against different versions of the HDF5 library. You might be able to confirm this with ldd. For example, figure out where your h5py is installed with

python3 -c "import h5py; print(h5py.__file__)"

and there should be a bunch of *.so files in the same directory. You can run ldd on one of those .so files and libisce3.so to see if they link to the same libhdf5.so file (or if it was statically linked then it won't show up). My usual workaround is to install h5py from source, manually making sure to link with the same HDF5 library I used to compile isce3.

The isce3 developers plan to refactor the code to avoid sharing HDF5 objects between the C++ and Python layers, but that effort will take some time.

bhawkins avatar May 22 '23 16:05 bhawkins 


python3 -c "import h5py; print(h5py.__file__)"
/python/lib/python3.9/site-packages/h5py/__init__.py



ldd /home/testUser/isce/lib64/libisce3.so
        linux-vdso.so.1 =>  (0x00007ffd03f41000)
        libgdal.so.31 => /home/testUser/gdal/lib/libgdal.so.31 (0x00007fb3d9492000)
        libpyre.so => /python/compile/lib64/libpyre.so (0x00007fb3db258000)
        libhdf5_cpp.so.200 => /home/testUser/gdal/lib/libhdf5_cpp.so.200 (0x00007fb3db1da000)
        libhdf5.so.200 => /home/testUser/gdal/lib/libhdf5.so.200 (0x00007fb3d9054000)
        libhdfs.so.0.0.0 => /home/testUser/gdal/lib/libhdfs.so.0.0.0 (0x00007fb3d8e41000)
        libcrypto.so.3 => /home/testUser/gdal/lib64/libcrypto.so.3 (0x00007fb3d89d9000)
        libcurl.so.4 => /home/testUser/gdal/lib/libcurl.so.4 (0x00007fb3db14d000)
        libsz.so.2 => /home/testUser/gdal/lib/libsz.so.2 (0x00007fb3db138000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fb3d87c3000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fb3d85bf000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fb3d82bd000)
        libjournal.so => /python/compile/lib64/libjournal.so (0x00007fb3db0a4000)
        libfftw3f.so.3 => /home/testUser/isce/lib/libfftw3f.so.3 (0x00007fb3d819d000)
        libfftw3f_threads.so.3 => /home/testUser/isce/lib/libfftw3f_threads.so.3 (0x00007fb3db09a000)
        libfftw3_threads.so.3 => /home/testUser/isce/lib/libfftw3_threads.so.3 (0x00007fb3db091000)
        libgomp.so.1 => /lib64/libgomp.so.1 (0x00007fb3d7f77000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb3d7d5b000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fb3d7be7000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb3d79d1000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fb3d7603000)
        libSFCGAL.so.1 => /home/testUser/gdal/lib64/libSFCGAL.so.1 (0x00007fb3d6b42000)
        libgeos_c.so.1 => /home/testUser/gdal/lib64/libgeos_c.so.1 (0x00007fb3d6afa000)
        libexpat.so.1 => /lib64/libexpat.so.1 (0x00007fb3d68cf000)
        libopenjp2.so.7 => /lib64/libopenjp2.so.7 (0x00007fb3d667c000)
        libnetcdf.so.19 => /home/testUser/gdal/lib/libnetcdf.so.19 (0x00007fb3d6427000)
        libpng15.so.15 => /lib64/libpng15.so.15 (0x00007fb3d61fc000)
        libpq.so.5 => /home/testUser/gdal/lib/libpq.so.5 (0x00007fb3d5fac000)
        liblz4.so.1 => /lib64/liblz4.so.1 (0x00007fb3d5d9d000)
        libzstd.so.1 => /home/testUser/gdal/lib/libzstd.so.1 (0x00007fb3d5cb5000)
        liblzma.so.5 => /home/testUser/gdal/lib/liblzma.so.5 (0x00007fb3d5c8d000)
        libproj.so.25 => /home/testUser/gdal/lib64/libproj.so.25 (0x00007fb3d588f000)
        libsqlite3.so.0 => /home/testUser/gdal/lib/libsqlite3.so.0 (0x00007fb3d574c000)
        libtiff.so.5 => /home/testUser/gdal/lib/libtiff.so.5 (0x00007fb3d56d3000)
        libjpeg.so.62 => /lib64/libjpeg.so.62 (0x00007fb3d547e000)
        libdeflate.so.0 => /home/testUser/gdal/lib/libdeflate.so.0 (0x00007fb3db077000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fb3d5276000)
        libpcre2-8.so.0 => /home/testUser/gdal/lib/libpcre2-8.so.0 (0x00007fb3d51da000)
        libiconv.so.2 => /home/testUser/gdal/lib/libiconv.so.2 (0x00007fb3d50f4000)
        libxml2.so.2 => /lib64/libxml2.so.2 (0x00007fb3d4d8a000)
        libjvm.so => /home/testUser/java/jdk1.8.0_211/jre/lib/amd64/server/libjvm.so (0x00007fb3d3da1000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb3db04d000)
        libssl.so.3 => /home/testUser/gdal/lib64/libssl.so.3 (0x00007fb3d3cf8000)
        libfftw3.so.3 => /home/testUser/isce/lib/libfftw3.so.3 (0x00007fb3d3bd1000)
        libgmpxx.so.4 => /lib64/libgmpxx.so.4 (0x00007fb3d39ca000)
        libboost_thread.so.1.80.0 => /home/testUser/gdal/lib/libboost_thread.so.1.80.0 (0x00007fb3d39af000)
        libboost_system.so.1.80.0 => /home/testUser/gdal/lib/libboost_system.so.1.80.0 (0x00007fb3d39aa000)
        libboost_serialization.so.1.80.0 => /home/testUser/gdal/lib/libboost_serialization.so.1.80.0 (0x00007fb3d3954000)
        libmpfr.so.6 => /home/testUser/gdal/lib/libmpfr.so.6 (0x00007fb3d36aa000)
        libgmp.so.10 => /home/testUser/gdal/lib/libgmp.so.10 (0x00007fb3d3633000)
        libgeos.so.3.11.0 => /home/testUser/gdal/lib64/libgeos.so.3.11.0 (0x00007fb3d3391000)
        libhdf5_hl.so.200 => /home/testUser/gdal/lib/libhdf5_hl.so.200 (0x00007fb3d336c000)
        libssl.so.10 => /lib64/libssl.so.10 (0x00007fb3d30fa000)
        libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fb3d2c97000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fb3d2a4a000)
        libldap_r-2.4.so.2 => /lib64/libldap_r-2.4.so.2 (0x00007fb3d27eb000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fb3d2502000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fb3d22fd000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fb3d20ca000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fb3d1eba000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fb3d1cb5000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fb3d1a9b000)
        liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007fb3d188c000)
        libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007fb3d166e000)
        libssl3.so => /lib64/libssl3.so (0x00007fb3d1409000)
        libsmime3.so => /lib64/libsmime3.so (0x00007fb3d11e1000)
        libnss3.so => /lib64/libnss3.so (0x00007fb3d0ea6000)
        libnssutil3.so => /lib64/libnssutil3.so (0x00007fb3d0c76000)
        libplds4.so => /lib64/libplds4.so (0x00007fb3d0a72000)
        libplc4.so => /lib64/libplc4.so (0x00007fb3d086c000)
        libnspr4.so => /lib64/libnspr4.so (0x00007fb3d062d000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fb3d0405000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fb3d01ce000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fb3cff6b000)
        libfreebl3.so => /lib64/libfreebl3.so (0x00007fb3cfd68000)

zyxgis avatar May 29 '23 02:05 zyxgis

i compile hdf5-1.12.2.tar.gz i install h5py-3.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl to python3

zyxgis avatar May 29 '23 02:05 zyxgis

Still need to see the output of ldd on one of the .so files in /python/lib/python3.9/site-packages/h5py. My guess is that it will be using a different libhdf5.so than your isce3 library uses, which is the problem. If that's the case, my usual workaround is to install h5py from source (not a binary .whl).

bhawkins avatar May 31 '23 18:05 bhawkins