satpy icon indicating copy to clipboard operation
satpy copied to clipboard

Published 20 hours ago • verified publisher icon pytroll

Unable to open Data Tailor MSG SEVIRI Native file

Open jacobbieker opened this issue 2 years ago • 11 comments

Describe the bug I've downloaded from the data tailor service a subsetted MSG SEVIRI native file over the UK. As its still in the native format, I was expecting to be able to open it with the script below, but instead errors are thrown and it fails.

To Reproduce

hrv_scene = Scene(filenames={"seviri_l1b_native": ["/home/jacob/Development/Satip/scripts/MSG4-SEVI-MSG15-0100-NA-20220623152742.122000000Z-NA.nat"]})
print(hrv_scene.all_dataset_names())
hrv_scene.load(
    [
        "HRV",
    ],
)
print(hrv_scene)

Expected behavior The file to be opened and able to extract channels like with the full sized native files.

Actual results Text output of actual results or error messages including full tracebacks if applicable.

/home/jacob/anaconda3/envs/satip/bin/python /home/jacob/Development/Satip/scripts/data_tailor.py
[DEBUG: 2022-06-23 16:57:02 : satpy.readers.yaml_reader] Reading ('/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/etc/readers/seviri_l1b_native.yaml',)
/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/readers/seviri_base.py:453: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  ('GsicsCalMode', np.bool),
/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/readers/seviri_base.py:454: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  ('GsicsCalValidity', np.bool),
[DEBUG: 2022-06-23 16:57:02 : satpy.readers.yaml_reader] Assigning to seviri_l1b_native: ['/home/jacob/Development/Satip/scripts/MSG4-SEVI-MSG15-0100-NA-20220623152742.122000000Z-NA.nat']
['HRV', 'IR_016', 'IR_039', 'IR_087', 'IR_097', 'IR_108', 'IR_120', 'IR_134', 'VIS006', 'VIS008', 'WV_062', 'WV_073']
[DEBUG: 2022-06-23 16:57:02 : satpy.composites.config_loader] Looking for composites config file seviri.yaml
[DEBUG: 2022-06-23 16:57:02 : satpy.composites.config_loader] Looking for composites config file visir.yaml

[WARNING: 2022-06-23 16:57:02 : satpy.readers.yaml_reader] Failed to load DataID(name='HRV', wavelength=WavelengthRange(min=0.5, central=0.7, max=0.9, unit='µm'), resolution=1000.134348869, calibration=<calibration.reflectance>, modifiers=()) from <NativeMSGFileHandler: '/home/jacob/Development/Satip/scripts/MSG4-SEVI-MSG15-0100-NA-20220623152742.122000000Z-NA.nat'>
Traceback (most recent call last):
  File "/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/readers/yaml_reader.py", line 699, in _load_dataset
    projectable = fh.get_dataset(dsid, ds_info)
  File "/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/readers/seviri_l1b_native.py", line 454, in get_dataset
    raise KeyError('Channel % s not available in the file' % dataset_id['name'])
KeyError: 'Channel HRV not available in the file'
[ERROR: 2022-06-23 16:57:02 : satpy.readers.yaml_reader] Could not load dataset 'DataID(name='HRV', wavelength=WavelengthRange(min=0.5, central=0.7, max=0.9, unit='µm'), resolution=1000.134348869, calibration=<calibration.reflectance>, modifiers=())': "Could not load DataID(name='HRV', wavelength=WavelengthRange(min=0.5, central=0.7, max=0.9, unit='µm'), resolution=1000.134348869, calibration=<calibration.reflectance>, modifiers=()) from any provided files"
Traceback (most recent call last):
  File "/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/readers/yaml_reader.py", line 851, in _load_dataset_with_area
    ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
  File "/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/readers/yaml_reader.py", line 723, in _load_dataset_data
    proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
  File "/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/readers/yaml_reader.py", line 708, in _load_dataset
    raise KeyError(
KeyError: "Could not load DataID(name='HRV', wavelength=WavelengthRange(min=0.5, central=0.7, max=0.9, unit='µm'), resolution=1000.134348869, calibration=<calibration.reflectance>, modifiers=()) from any provided files"
[WARNING: 2022-06-23 16:57:02 : satpy.scene] The following datasets were not created and may require resampling to be generated: DataID(name='HRV', wavelength=WavelengthRange(min=0.5, central=0.7, max=0.9, unit='µm'), resolution=1000.134348869, calibration=<calibration.reflectance>, modifiers=())

Screenshots If applicable, add screenshots to help explain your problem.

Environment Info:

  • OS: Linux
  • Satpy Version: 0.34.0
  • PyResample Version: 1.22.3
  • Readers and writers dependencies (when relevant):
Readers
=======
/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/readers/seviri_base.py:453: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
 ('GsicsCalMode', np.bool),
/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/readers/seviri_base.py:454: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
 ('GsicsCalValidity', np.bool),
abi_l1b:  ok
abi_l1b_scmi:  ok
abi_l2_nc:  ok
acspo:  ok
agri_l1:  ok
ahi_hrit:  ok
ahi_hsd:  ok
ahi_l1b_gridded_bin:  ok
ami_l1b:  ok
amsr2_l1b:  ok
amsr2_l2:  ok
amsr2_l2_gaasp:  ok
amsub_l1c_aapp:  ok
ascat_l2_soilmoisture_bufr:  ok
avhrr_l1b_aapp:  ok
avhrr_l1b_eps:  ok
avhrr_l1b_gaclac:  cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac')
avhrr_l1b_hrpt:  ok
avhrr_l1c_eum_gac_fdr_nc:  ok
caliop_l2_cloud:  cannot find module 'satpy.readers.caliop_l2_cloud' (No module named 'pyhdf')
clavrx:  cannot find module 'satpy.readers.clavrx' (No module named 'pyhdf')
cmsaf-claas2_l2_nc:  ok
electrol_hrit:  ok
fci_l1c_nc:  ok
fci_l2_nc:  ok
generic_image:  ok
geocat:  ok
ghrsst_l3c_sst:  cannot find module 'satpy.readers.ghrsst_l3c_sst' (cannot import name 'Dataset' from 'satpy.dataset' (/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/dataset/__init__.py))
glm_l2:  ok
goes-imager_hrit:  ok
goes-imager_nc:  ok
gpm_imerg:  ok
grib:  cannot find module 'satpy.readers.grib' (No module named 'pygrib')
hsaf_grib:  cannot find module 'satpy.readers.hsaf_grib' (No module named 'pygrib')
hy2_scat_l2b_h5:  ok
iasi_l2:  ok
iasi_l2_so2_bufr:  ok
jami_hrit:  ok
li_l2:  cannot find module 'satpy.readers.li_l2' (cannot import name 'Dataset' from 'satpy.dataset' (/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/dataset/__init__.py))
maia:  ok
mersi2_l1b:  ok
mhs_l1c_aapp:  ok
mimicTPW2_comp:  ok
mirs:  ok
modis_l1b:  cannot find module 'satpy.readers.modis_l1b' (No module named 'pyhdf')
modis_l2:  cannot find module 'satpy.readers.modis_l2' (No module named 'pyhdf')
msi_safe:  ok
msu_gsa_l1b:  ok
mtsat2-imager_hrit:  ok
mviri_l1b_fiduceo_nc:  ok
nucaps:  ok
nwcsaf-geo:  ok
nwcsaf-msg2013-hdf5:  ok
nwcsaf-pps_nc:  ok
olci_l1b:  ok
olci_l2:  ok
omps_edr:  ok
safe_sar_l2_ocn:  ok
sar-c_safe:  cannot find module 'satpy.readers.sar_c_safe' (No module named 'defusedxml')
satpy_cf_nc:  ok
scatsat1_l2b:  cannot find module 'satpy.readers.scatsat1_l2b' (cannot import name 'Dataset' from 'satpy.dataset' (/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/dataset/__init__.py))
seadas_l2:  cannot find module 'satpy.readers.seadas_l2' (No module named 'pyhdf')
seviri_l1b_hrit:  ok
seviri_l1b_icare:  cannot find module 'satpy.readers.seviri_l1b_icare' (No module named 'pyhdf')
seviri_l1b_native:  ok
seviri_l1b_nc:  ok
seviri_l2_bufr:  ok
seviri_l2_grib:  ok
slstr_l1b:  ok
slstr_l2:  ok
smos_l2_wind:  ok
tropomi_l2:  ok
vaisala_gld360:  ok
vii_l1b_nc:  ok
vii_l2_nc:  ok
viirs_compact:  ok
viirs_edr_active_fires:  ok
viirs_edr_flood:  cannot find module 'satpy.readers.viirs_edr_flood' (No module named 'pyhdf')
viirs_l1b:  ok
viirs_sdr:  ok
virr_l1b:  ok

Writers
=======
/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/writers/cf_writer.py:143: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
 from distutils.version import LooseVersion
awips_tiled:  ok
cf:  ok
geotiff:  ok
mitiff:  ok
ninjogeotiff:  ok
ninjotiff:  cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff')
simple_image:  ok

Extras
======
cartopy:  ok
geoviews:  No module named 'geoviews'

Additional context I can open the full sized native files fine with this installation.

jacobbieker avatar Jun 23 '22 16:06 jacobbieker

I haven't tested any data subsetted with data tailor, but I'd also expect it to work. Does any of the other channels load? The HRV channel is a bit special being composed of two separate imaging windows, so I wouldn't be surprised if the data tailor and/or Satpy would behave a bit unexpectedly.

pnuu avatar Jun 23 '22 16:06 pnuu

Yeah, it works loading the other channels, just not HRV

jacobbieker avatar Jun 23 '22 16:06 jacobbieker

Here is the file I am using: https://drive.google.com/file/d/1yNhQt-M6IxJV7ia4XHBPGlC-YzKD3yaC/view?usp=sharing if that's helpful at all

jacobbieker avatar Jun 23 '22 16:06 jacobbieker

Also, it doesn't seem to work with NetCDF files either.

jacobbieker avatar Jul 08 '22 11:07 jacobbieker

Taking a look at the Data Tailor documentation, this might be related to a limitation of the Data Tailor itself, see: https://eumetsatspace.atlassian.net/wiki/spaces/DSDT/pages/1558872065/DTWS+and+DT+standalone+Customisation+of+HRV+channel+MSG+data
It seems that the HRV channel needs to be customised separately. If you haven't done it yet, could you try if it works with data produced as suggested in the documentation?

ameraner avatar Jul 08 '22 13:07 ameraner

Yeah, I tried that, the option for NetCDF doesn't work with SatPy, and there isn't a native file option for just HRV.

Trying it with this customized NetCDF file: https://drive.google.com/file/d/17AixEstjPj6TAGn5dMi7k3D459vOXIm9/view?usp=sharing

hrv_scene = Scene(filenames={"seviri_l1b_nc": ["/home/jacob/Development/Satip/scripts/HRSEVIRI_HRV_20220708T134510Z_20220708T134510Z_epct_7dbd8ef9_C.nc"],}) print(hrv_scene.all_dataset_names()) print(hrv_scene.all_dataset_ids()) hrv_scene.load( [ 'HRV' ], )

I get No filenames found for reader: seviri_l1b_nc Don't know how to open the following files: {'/home/jacob/Development/Satip/scripts/HRSEVIRI_HRV_20220708T134510Z_20220708T134510Z_epct_7dbd8ef9_C.nc'} Traceback (most recent call last): File "/home/jacob/Development/Satip/scripts/data_tailor.py", line 38, in <module> hrv_scene = Scene(filenames={"seviri_l1b_nc": ["/home/jacob/Development/Satip/scripts/HRSEVIRI_HRV_20220708T134510Z_20220708T134510Z_epct_7dbd8ef9_C.nc"],}) File "/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/scene.py", line 121, in __init__ self._readers = self._create_reader_instances(filenames=filenames, File "/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/scene.py", line 142, in _create_reader_instances return load_readers(filenames=filenames, File "/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/readers/__init__.py", line 578, in load_readers _check_reader_instances(reader_instances) File "/home/jacob/anaconda3/envs/satip/lib/python3.10/site-packages/satpy/readers/__init__.py", line 617, in _check_reader_instances raise ValueError("No supported files found") ValueError: No supported files found

jacobbieker avatar Jul 08 '22 14:07 jacobbieker

It does work to open Data Tailor HRIT files though, and just the subsetted rows of the HRIT data that I want, so I think that works well enough for my use case.

jacobbieker avatar Jul 10 '22 18:07 jacobbieker

I took another look at this:

  • native: the output of the get_available_channels here https://github.com/pytroll/satpy/blob/15dbf3db78b0f23aa2c8b1acda458212dfca0e84/satpy/readers/seviri_l1b_native.py#L769 for the file that you have linked above is {'VIS006': True, 'VIS008': True, 'IR_016': True, 'IR_039': True, 'WV_062': True, 'WV_073': True, 'IR_087': True, 'IR_097': True, 'IR_108': True, 'IR_120': True, 'IR_134': True, 'HRV': False} which indicates that HRV is not included in the file. Just in case they just forgot to update the header, I've also tried to manually set it to True, but it doesn't help. It seems that this is an issue on the Data Tailor side, if it's not possible to produce a native file containing the HRV.
  • NetCDF: the error you are seeing comes from the fact that Satpy does not recognise the filename, as it is different to the common SEVIRI NetCDF filenaming. But, regardless of that, I took a look at the file you linked above and the content is completely different compared to a normal SEVIRI NetCDF. So, even after fixing the filenaming recognition, Satpy would not work because the Data Tailor output format is different. So, bad news... we need a new, customised reader to be able to read this type of data with Satpy. If you have time and are motivated, feel free to propose a first draft :)

ameraner avatar Jul 11 '22 16:07 ameraner

Ah okay, thanks for investigating it more! I guess once they fix the limitation of separate non-HRV and HRV channels it should then start working. For the NetCDF issue, because the HRIT files seem to be working fine, and the native file might get fixed soon on EUMETSAT's side, I'll probably leave it for now, but might come back to it if it turns out we need it.

jacobbieker avatar Jul 11 '22 18:07 jacobbieker

Thanks! Regarding the Data Tailor not working properly with HRV, have you notified EUMETSAT about this already, or are you aware if they are actively working on this? Otherwise, if you have the chance, I would suggest to report it with an email to [email protected].

ameraner avatar Jul 12 '22 08:07 ameraner

Hi all, just looking at this. Is the issue about the existence of HRV data?

I have a local installation of the data tailor and while the data (non-HRV and HRV) is in the file, you still need to call the data tailor (epct at the command-line) twice, one for the non-HRV (HRSEVIRI product) and one for the HRV (HRSEVIRI_HRV product). See https://eumetsatspace.atlassian.net/wiki/spaces/DSDT/pages/1854078977/Installation+instructions+V3.0.0+onwards

epct run-chain -f filter-hrit.yaml -o MSG4_20220519/HRIT_202205191100 MSG4-SEVI-MSG15-0100-NA-20220519111241.948000000Z-NA.zip

with the file filter-hrit.yaml

format: hrit
product: HRSEVIRI

(repeat with HRSEVIRI_HRV).

The input file is simply downloaded from their data store and it is the same file containing all output.

In case this was not the issue, please forget the message :-)

pdebuyl avatar Aug 09 '22 14:08 pdebuyl