pygmtsar
pygmtsar copied to clipboard
AlexeyPechnikov
[Help]: Error in interferogram creation
I'm trying to create interferograms (sbas.intf_parallel(pairs, wavelength=60, func=intf_decimator) but I don't know why it's giving me the following error:
"""
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 209, in _acquire_with_cache_info
file = self._cache[self._key]
File "/home/ubuntu/.local/lib/python3.8/site-packages/xarray/backends/lru_cache.py", line 55, in __getitem__
value = self._cache[key]
KeyError: [<class 'h5netcdf.core.File'>, ('/home/ubuntu/dsc_sul/A7_asc/A7_raw_asc/F2_20150107_20150119_realfilt.grd',), 'r', (('decode_vlen_strings', True), ('invalid_netcdf', None)), 'a11b417a-a84f-499d-bcd5-80afac972c62']
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 463, in _process_worker
r = call_item()
File "/home/ubuntu/.local/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 291, in __call__
return self.fn(*self.args, **self.kwargs)
File "/home/ubuntu/.local/lib/python3.8/site-packages/joblib/parallel.py", line 589, in __call__
return [func(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.8/site-packages/joblib/parallel.py", line 589, in <listcomp>
return [func(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.8/site-packages/pygmtsar/SBAS_intf.py", line 18, in intf
prm_ref.intf(prm_rep,
File "/home/ubuntu/.local/lib/python3.8/site-packages/pygmtsar/PRM.py", line 516, in intf
realfilt = xr.open_dataarray(fullname('realfilt.grd'), engine=self.engine, chunks=chunksize)
File "/home/ubuntu/.local/lib/python3.8/site-packages/xarray/backends/api.py", line 702, in open_dataarray
dataset = open_dataset(
File "/home/ubuntu/.local/lib/python3.8/site-packages/xarray/backends/api.py", line 541, in open_dataset
backend_ds = backend.open_dataset(
File "/home/ubuntu/.local/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py", line 407, in open_dataset
store = H5NetCDFStore.open(
File "/home/ubuntu/.local/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py", line 170, in open
return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
File "/home/ubuntu/.local/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py", line 120, in __init__
self._filename = find_root_and_group(self.ds)[0].filename
File "/home/ubuntu/.local/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py", line 181, in ds
return self._acquire()
File "/home/ubuntu/.local/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py", line 173, in _acquire
with self._manager.acquire_context(needs_lock) as root:
File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/home/ubuntu/.local/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 197, in acquire_context
file, cached = self._acquire_with_cache_info(needs_lock)
File "/home/ubuntu/.local/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 215, in _acquire_with_cache_info
file = self._opener(*self._args, **kwargs)
File "/home/ubuntu/.local/lib/python3.8/site-packages/h5netcdf/core.py", line 1052, in __init__
self._h5file = self._h5py.File(
File "/home/ubuntu/.local/lib/python3.8/site-packages/h5py/_hl/files.py", line 562, in __init__
fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
File "/home/ubuntu/.local/lib/python3.8/site-packages/h5py/_hl/files.py", line 235, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 102, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to synchronously open file (unable to open file: name = '/home/ubuntu/dsc_sul/A7_asc/A7_raw_asc/F2_20150107_20150119_realfilt.grd', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
"""
What do you think is the cause of this?
Python 3.8 is quite outdated and is not compatible with many Python libraries, including PyGMTSAR itself. It would be advisable to use Python 3.10 or 3.11 instead.
I already tried using 3.11 version of python but still having the same error:
`""" Traceback (most recent call last): File "/home/ubuntu/miniconda3/envs/pygmtsar/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info file = self._cache[self._key] ~~~~~~~~~~~^^^^^^^^^^^ File "/home/ubuntu/miniconda3/envs/pygmtsar/lib/python3.11/site-packages/xarray/backends/lru_cache.py", line 56, in getitem value = self._cache[key] ~~~~~~~~~~~^^^^^ KeyError: [<class 'h5netcdf.core.File'>, ('/home/ubuntu/dsc_sul/A7_asc/A7_raw_asc/F2_20150119_20150131_realfilt.grd',), 'r', (('decode_vlen_strings', True), ('driver', None), ('invalid_netcdf', None)), '1ad9ab81-3976-4c92-b9fa-28fdcc6d8f6c']
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/envs/pygmtsar/lib/python3.11/site-packages/joblib/externals/loky/process_executor.py", line 463, in process_worker
r = call_item()
^^^^^^^^^^^
File "/home/ubuntu/miniconda3/envs/pygmtsar/lib/python3.11/site-packages/joblib/externals/loky/process_executor.py", line 291, in call
return self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/miniconda3/envs/pygmtsar/lib/python3.11/site-packages/joblib/parallel.py", line 589, in call
return [func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/miniconda3/envs/pygmtsar/lib/python3.11/site-packages/joblib/parallel.py", line 589, in
The above exception was the direct cause of the following exception:
FileNotFoundError Traceback (most recent call last) Cell In[18], line 1 ----> 1 sbas.intf_parallel(pairs, wavelength=60, func=intf_decimator)
File ~/miniconda3/envs/pygmtsar/lib/python3.11/site-packages/pygmtsar/SBAS_intf.py:58, in SBAS_intf.intf_parallel(self, pairs, n_jobs, **kwargs)
55 loky.get_reusable_executor(kill_workers=True).shutdown(wait=True)
56 with joblib.parallel_backend('loky', n_jobs=n_jobs):
57 # convert string subswath to integer value
---> 58 joblib.Parallel()(joblib.delayed(self.intf)(int(subswath), [date1, date2], **kwargs)
59 for (subswath,date1,date2) in chunk)
60 pbar.update(len(chunk))
File ~/miniconda3/envs/pygmtsar/lib/python3.11/site-packages/joblib/parallel.py:1952, in Parallel.call(self, iterable)
1946 # The first item from the output is blank, but it makes the interpreter
1947 # progress until it enters the Try/Except block of the generator and
1948 # reach the first yield statement. This starts the aynchronous
1949 # dispatch of the tasks to the workers.
1950 next(output)
-> 1952 return output if self.return_generator else list(output)
File ~/miniconda3/envs/pygmtsar/lib/python3.11/site-packages/joblib/parallel.py:1595, in Parallel._get_outputs(self, iterator, pre_dispatch) 1592 yield 1594 with self._backend.retrieval_context(): -> 1595 yield from self._retrieve() 1597 except GeneratorExit: 1598 # The generator has been garbage collected before being fully 1599 # consumed. This aborts the remaining tasks if possible and warn 1600 # the user if necessary. 1601 self._exception = True
File ~/miniconda3/envs/pygmtsar/lib/python3.11/site-packages/joblib/parallel.py:1699, in Parallel._retrieve(self)
1692 while self._wait_retrieval():
1693
1694 # If the callback thread of a worker has signaled that its task
1695 # triggered an exception, or if the retrieval loop has raised an
1696 # exception (e.g. GeneratorExit), exit the loop and surface the
1697 # worker traceback.
1698 if self._aborting:
-> 1699 self._raise_error_fast()
1700 break
1702 # If the next job is not ready for retrieval yet, we just wait for
1703 # async callbacks to progress.
File ~/miniconda3/envs/pygmtsar/lib/python3.11/site-packages/joblib/parallel.py:1734, in Parallel._raise_error_fast(self) 1730 # If this error job exists, immediatly raise the error by 1731 # calling get_result. This job might not exists if abort has been 1732 # called directly or if the generator is gc'ed. 1733 if error_job is not None: -> 1734 error_job.get_result(self.timeout)
File ~/miniconda3/envs/pygmtsar/lib/python3.11/site-packages/joblib/parallel.py:736, in BatchCompletionCallBack.get_result(self, timeout) 730 backend = self.parallel._backend 732 if backend.supports_retrieve_callback: 733 # We assume that the result has already been retrieved by the 734 # callback thread, and is stored internally. It's just waiting to 735 # be returned. --> 736 return self._return_or_raise() 738 # For other backends, the main thread needs to run the retrieval step. 739 try:
File ~/miniconda3/envs/pygmtsar/lib/python3.11/site-packages/joblib/parallel.py:754, in BatchCompletionCallBack._return_or_raise(self) 752 try: 753 if self.status == TASK_ERROR: --> 754 raise self._result 755 return self._result 756 finally:
FileNotFoundError: [Errno 2] Unable to synchronously open file (unable to open file: name = '/home/ubuntu/dsc_sul/A7_asc/A7_raw_asc/F2_20150119_20150131_realfilt.grd', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)`
What is your environment? The error could be related to your attempt to use write locks on a network file system.
By the way, you can try your processing in a Docker container. Use Dockerfile in the repository to build it.
Hi Alexey, I've tried using your docker container with python=3.10 and 3.11 for creating interferograms and I keep getting the same error, I can't figure out why...
Hi Ana, please use the new PyGMTSAR version released a few days ago. It includes PSI and SBAS analyses and has an AI Virtual Assistant. The old version should work as well (as it has been tested on Google Colab), but it may encounter unpredictable conflicts with new Python libraries on different systems.