satpy icon indicating copy to clipboard operation
satpy copied to clipboard
verified publisher icon pytroll

process time increase from 0.36.0 to 0.37.0 when streaming netcdf over opendap

Open TAlonglong opened this issue 3 years ago • 16 comments

Feature Request

This is neither a bug nor a feature I think, but anyway.

So I have a simple script to open, load, resample and save a composite png to file with data from a thredds server with opendap.

import sys
from satpy import Scene
from satpy.utils import debug_on
debug_on()

file_in = sys.argv[1]

global_scene = Scene(reader='satpy_cf_nc', filenames=[file_in])
print(global_scene.available_dataset_names())
print(global_scene.all_composite_names() )

global_scene.load(['hr_overview'])

l = global_scene.resample('euro4')
l.save_dataset('hr_overview', f"hr_overview-{l.start_time:%Y%m%d-%H%M%S}.png")

If run like this python test-thredds-satpy-netcdf.py https://thredds.met.no/thredds/dodsC/remotesensingsatellite/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc with satpy 0.36.0 I get a run time of aprox 4 minutes: Complete log:

/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
[DEBUG: 2022-08-19 09:28:15 : satpy.readers.yaml_reader] Reading ('/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/satpy/etc/readers/satpy_cf_nc.yaml',)
[DEBUG: 2022-08-19 09:28:15 : satpy.readers.yaml_reader] Assigning to satpy_cf_nc: ['https://thredds.met.no/thredds/dodsC/remotesensingsatellite/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc']
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/cfgrib/xarray_plugin.py:10: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(xr.__version__) <= "0.17.0":
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
['I01', 'I02', 'I03', 'I04', 'I05', 'latitude', 'longitude', 'satellite_azimuth_angle', 'satellite_zenith_angle', 'solar_azimuth_angle', 'solar_zenith_angle']
[DEBUG: 2022-08-19 09:28:16 : satpy.composites.config_loader] Looking for composites config file viirs.yaml
[DEBUG: 2022-08-19 09:28:16 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2022-08-19 09:28:16 : satpy.node] Skipping optional DataQuery(wavelength=13.4, calibration='brightness_temperature'): Unknown dataset DataQuery(wavelength=13.4, calibration='brightness_temperature')
['cloudtop', 'green_snow', 'hr_cloudtop_daytime', 'hr_overview', 'ir108_3d', 'ir_cloud_day', 'natural_color_raw', 'natural_color_sun', 'snow', 'snow_hires', 'ssec_fog']
[DEBUG: 2022-08-19 09:28:16 : satpy.readers.satpy_cf_nc] Getting data for: I02
[DEBUG: 2022-08-19 09:28:16 : satpy.readers.satpy_cf_nc] Getting data for: I05
[DEBUG: 2022-08-19 09:28:16 : satpy.readers.satpy_cf_nc] Getting data for: I01
[DEBUG: 2022-08-19 09:28:16 : satpy.scene] Unloading dataset: DataID(name='I02', wavelength=WavelengthRange(min=0.845, central=0.865, max=0.884, unit='µm'), resolution=371, calibration=<calibration.reflectance>, modifiers=('sunz_corrected_iband',))
[DEBUG: 2022-08-19 09:28:16 : satpy.scene] Unloading dataset: DataID(name='I05', wavelength=WavelengthRange(min=10.5, central=11.45, max=12.3, unit='µm'), resolution=371, calibration=<calibration.brightness_temperature>, modifiers=())
[DEBUG: 2022-08-19 09:28:16 : satpy.scene] Unloading dataset: DataID(name='I01', wavelength=WavelengthRange(min=0.6, central=0.64, max=0.68, unit='µm'), resolution=371, calibration=<calibration.reflectance>, modifiers=('sunz_corrected_iband',))
[DEBUG: 2022-08-19 09:28:17 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-19 09:28:17 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-19 09:28:17 : satpy.scene] Resampling DataID(name='hr_overview', resolution=371)
[INFO: 2022-08-19 09:28:17 : satpy.scene] Not reducing data before resampling.
[INFO: 2022-08-19 09:28:17 : satpy.resample] Using default KDTree resampler
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/pyresample/geometry.py:528: DeprecationWarning: This function is deprecated. See: https://pyproj4.github.io/pyproj/stable/gotchas.html#upgrading-to-pyproj-2-from-pyproj-1
  xyz = np.stack(transform(src, dst, lons, lats, alt), axis=1)
[DEBUG: 2022-08-19 09:28:17 : satpy.resample] Computing kd-tree parameters
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/pyresample/kd_tree.py:1025: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  input_coords = input_coords.astype(np.float)
[DEBUG: 2022-08-19 09:28:17 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-19 09:28:17 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/pyresample/kd_tree.py:1047: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  radius=self.radius_of_influence, dtype=np.int,
[DEBUG: 2022-08-19 09:28:17 : satpy.resample] Resampling where-f7bb1fe9793d87ea6c56d9025555694e
[DEBUG: 2022-08-19 09:28:17 : satpy.writers] Reading ['/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/satpy/etc/writers/simple_image.yaml']
[DEBUG: 2022-08-19 09:28:17 : satpy.writers] Enhancement configuration options: [{'name': 'inverse', 'method': <function invert at 0x7f7cff189790>, 'args': [[False, False, True]]}, {'name': 'stretch', 'method': <function stretch at 0x7f7cff189d30>, 'kwargs': {'stretch': 'linear'}}, {'name': 'gamma', 'method': <function gamma at 0x7f7cff189e50>, 'kwargs': {'gamma': 1.6}}]
[DEBUG: 2022-08-19 09:28:17 : trollimage.xrimage] Applying invert with parameters [False, False, True]
[DEBUG: 2022-08-19 09:28:17 : trollimage.xrimage] Applying stretch linear with parameters {}
[DEBUG: 2022-08-19 09:28:17 : trollimage.xrimage] Perform a linear contrast stretch.
[DEBUG: 2022-08-19 09:28:17 : trollimage.xrimage] Calculate the histogram quantiles: 
[DEBUG: 2022-08-19 09:28:17 : trollimage.xrimage] Left and right quantiles: 0.005 0.005
[DEBUG: 2022-08-19 09:28:17 : trollimage.xrimage] Applying gamma 1.6
[DEBUG: 2022-08-19 09:28:17 : satpy.writers.simple_image] Saving to image: hr_overview-20220818-102458.png
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing BlpImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing BmpImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing BufrStubImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing CurImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing DcxImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing DdsImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing EpsImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing FitsStubImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing FliImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing FpxImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing FtexImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing GbrImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing GifImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing GribStubImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing Hdf5StubImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing IcnsImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing IcoImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing ImImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing ImtImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing IptcImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing JpegImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing Jpeg2KImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing McIdasImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing MicImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing MpegImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing MpoImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing MspImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing PalmImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing PcdImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing PcxImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing PdfImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing PixarImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing PngImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing PpmImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing PsdImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing SgiImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing SpiderImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing SunImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing TgaImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing TiffImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing WebPImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing WmfImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing XbmImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing XpmImagePlugin
[DEBUG: 2022-08-19 09:28:17 : PIL.Image] Importing XVThumbImagePlugin
[DEBUG: 2022-08-19 09:28:17 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-19 09:32:06 : trollimage.xrimage] Interval: left=<xarray.DataArray (bands: 3)>
array([   3.28414226,    1.47481191, -301.93130341])
Coordinates:
    quantile  float64 0.005
Dimensions without coordinates: bands, right=<xarray.DataArray (bands: 3)>
array([  87.22452698,   98.27553253, -224.84414673])
Coordinates:
    quantile  float64 0.995
Dimensions without coordinates: bands

But if I update satpy to 0.37.0 and install needed module aiohttp the run time is much longer, approx 15 minutes. Full log:

  "class": algorithms.Blowfish,
[DEBUG: 2022-08-19 07:55:00 : asyncio] Using selector: EpollSelector
[DEBUG: 2022-08-19 07:55:00 : satpy.readers.yaml_reader] Reading ('/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/satpy/etc/readers/satpy_cf_nc.yaml',)
[DEBUG: 2022-08-19 07:55:00 : satpy.readers.yaml_reader] Assigning to satpy_cf_nc: [<FSFile "https://thredds.met.no/thredds/dodsC/remotesensingsatellite/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc">]
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/cfgrib/xarray_plugin.py:10: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(xr.__version__) <= "0.17.0":
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
['I01', 'I02', 'I03', 'I04', 'I05', 'latitude', 'longitude', 'satellite_azimuth_angle', 'satellite_zenith_angle', 'solar_azimuth_angle', 'solar_zenith_angle']
[DEBUG: 2022-08-19 07:55:01 : satpy.composites.config_loader] Looking for composites config file viirs.yaml
[DEBUG: 2022-08-19 07:55:01 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2022-08-19 07:55:01 : satpy.node] Skipping optional DataQuery(wavelength=13.4, calibration='brightness_temperature'): Unknown dataset DataQuery(wavelength=13.4, calibration='brightness_temperature')
['cloudtop', 'green_snow', 'hr_cloudtop_daytime', 'hr_overview', 'ir108_3d', 'ir_cloud_day', 'natural_color_raw', 'natural_color_sun', 'snow', 'snow_hires', 'ssec_fog']
[DEBUG: 2022-08-19 07:55:01 : satpy.readers.satpy_cf_nc] Getting data for: I05
[DEBUG: 2022-08-19 07:55:01 : satpy.readers.satpy_cf_nc] Getting data for: I01
[DEBUG: 2022-08-19 07:55:01 : satpy.readers.satpy_cf_nc] Getting data for: I02
[DEBUG: 2022-08-19 08:07:03 : satpy.scene] Unloading dataset: DataID(name='I05', wavelength=WavelengthRange(min=10.5, central=11.45, max=12.3, unit='µm'), resolution=371, calibration=<calibration.brightness_temperature>, modifiers=())
[DEBUG: 2022-08-19 08:07:03 : satpy.scene] Unloading dataset: DataID(name='I01', wavelength=WavelengthRange(min=0.6, central=0.64, max=0.68, unit='µm'), resolution=371, calibration=<calibration.reflectance>, modifiers=('sunz_corrected_iband',))
[DEBUG: 2022-08-19 08:07:03 : satpy.scene] Unloading dataset: DataID(name='I02', wavelength=WavelengthRange(min=0.845, central=0.865, max=0.884, unit='µm'), resolution=371, calibration=<calibration.reflectance>, modifiers=('sunz_corrected_iband',))
[DEBUG: 2022-08-19 08:07:03 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-19 08:07:04 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-19 08:07:04 : satpy.scene] Resampling DataID(name='hr_overview', resolution=371)
[INFO: 2022-08-19 08:07:04 : satpy.scene] Not reducing data before resampling.
[INFO: 2022-08-19 08:07:04 : satpy.resample] Using default KDTree resampler
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/pyresample/geometry.py:528: DeprecationWarning: This function is deprecated. See: https://pyproj4.github.io/pyproj/stable/gotchas.html#upgrading-to-pyproj-2-from-pyproj-1
  xyz = np.stack(transform(src, dst, lons, lats, alt), axis=1)
[DEBUG: 2022-08-19 08:07:04 : satpy.resample] Computing kd-tree parameters
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/pyresample/kd_tree.py:1025: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  input_coords = input_coords.astype(np.float)
[DEBUG: 2022-08-19 08:07:04 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-19 08:07:04 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/pyresample/kd_tree.py:1047: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  radius=self.radius_of_influence, dtype=np.int,
[DEBUG: 2022-08-19 08:07:04 : satpy.resample] Resampling where-29a2718563646b4c43274efaf4d87ad3
[DEBUG: 2022-08-19 08:07:04 : satpy.writers] Reading ['/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/satpy/etc/writers/simple_image.yaml']
[DEBUG: 2022-08-19 08:07:04 : satpy.writers] Enhancement configuration options: [{'name': 'inverse', 'method': <function invert at 0x7f2a49920f70>, 'args': [[False, False, True]]}, {'name': 'stretch', 'method': <function stretch at 0x7f29e0526ca0>, 'kwargs': {'stretch': 'linear'}}, {'name': 'gamma', 'method': <function gamma at 0x7f29e0526dc0>, 'kwargs': {'gamma': 1.6}}]
[DEBUG: 2022-08-19 08:07:04 : trollimage.xrimage] Applying invert with parameters [False, False, True]
[DEBUG: 2022-08-19 08:07:04 : trollimage.xrimage] Applying stretch linear with parameters {}
[DEBUG: 2022-08-19 08:07:04 : trollimage.xrimage] Perform a linear contrast stretch.
[DEBUG: 2022-08-19 08:07:04 : trollimage.xrimage] Calculate the histogram quantiles: 
[DEBUG: 2022-08-19 08:07:04 : trollimage.xrimage] Left and right quantiles: 0.005 0.005
[DEBUG: 2022-08-19 08:07:04 : trollimage.xrimage] Applying gamma 1.6
[DEBUG: 2022-08-19 08:07:04 : satpy.writers.simple_image] Saving to image: hr_overview-20220818-102458.png
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing BlpImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing BmpImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing BufrStubImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing CurImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing DcxImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing DdsImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing EpsImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing FitsStubImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing FliImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing FpxImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing FtexImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing GbrImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing GifImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing GribStubImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing Hdf5StubImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing IcnsImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing IcoImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing ImImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing ImtImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing IptcImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing JpegImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing Jpeg2KImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing McIdasImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing MicImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing MpegImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing MpoImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing MspImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing PalmImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing PcdImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing PcxImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing PdfImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing PixarImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing PngImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing PpmImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing PsdImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing SgiImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing SpiderImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing SunImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing TgaImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing TiffImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing WebPImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing WmfImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing XbmImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing XpmImagePlugin
[DEBUG: 2022-08-19 08:07:04 : PIL.Image] Importing XVThumbImagePlugin
[DEBUG: 2022-08-19 08:07:04 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-19 08:10:42 : trollimage.xrimage] Interval: left=<xarray.DataArray (bands: 3)>
array([   3.28414226,    1.47481191, -301.93130341])
Coordinates:
    quantile  float64 0.005
Dimensions without coordinates: bands, right=<xarray.DataArray (bands: 3)>
array([  87.22452698,   98.27553253, -224.84414673])
Coordinates:
    quantile  float64 0.995
Dimensions without coordinates: bands

So there is a change in how satpy handles netcdf over opendap from 0.36.0 to 0.37.0 imacting the total process time. I got some hint this is due to the use of fsspec, and a possible solution is to make some special handling of opendap.

Is this possible?

TAlonglong avatar Aug 19 '22 07:08 TAlonglong

I might add both 0.36.0 and 0.37.0 produce the same image hr_overview-20220818-102458

0.37.0 just uses longer time.

TAlonglong avatar Aug 19 '22 07:08 TAlonglong

Is it only slow for the remote streamed file or is local file processing slower too? What about loading a single band? What about doing .compute() on the DataArray instead of saving it to disk? Put another way, how much of this example script can we take away and get similar performance. Has anything change in the reader(s) between the versions?

djhoese avatar Aug 19 '22 15:08 djhoese

0.36.0 local file ( i.e. over nfs mounted disk)

time python test-thredds-satpy-netcdf.py /lustre/storeB/immutable/archive/projects/remotesensing/satellite-thredds/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc 
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
[DEBUG: 2022-08-22 09:44:41 : satpy.readers.yaml_reader] Reading ('/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/satpy/etc/readers/satpy_cf_nc.yaml',)
[DEBUG: 2022-08-22 09:44:41 : satpy.readers.yaml_reader] Assigning to satpy_cf_nc: ['/lustre/storeB/immutable/archive/projects/remotesensing/satellite-thredds/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc']
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/cfgrib/xarray_plugin.py:10: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(xr.__version__) <= "0.17.0":
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/satpy/readers/satpy_cf_nc.py:240: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.
  if 'modifiers' in ds_info and not ds_info['modifiers']:
['I01', 'I02', 'I03', 'I04', 'I05', 'latitude', 'longitude', 'satellite_azimuth_angle', 'satellite_zenith_angle', 'solar_azimuth_angle', 'solar_zenith_angle']
[DEBUG: 2022-08-22 09:44:41 : satpy.composites.config_loader] Looking for composites config file viirs.yaml
[DEBUG: 2022-08-22 09:44:42 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2022-08-22 09:44:42 : satpy.node] Skipping optional DataQuery(wavelength=13.4, calibration='brightness_temperature'): Unknown dataset DataQuery(wavelength=13.4, calibration='brightness_temperature')
['cloudtop', 'green_snow', 'hr_cloudtop_daytime', 'hr_overview', 'ir108_3d', 'ir_cloud_day', 'natural_color_raw', 'natural_color_sun', 'snow', 'snow_hires', 'ssec_fog']
[DEBUG: 2022-08-22 09:44:42 : satpy.readers.satpy_cf_nc] Getting data for: I02
[DEBUG: 2022-08-22 09:44:42 : satpy.readers.satpy_cf_nc] Getting data for: I01
[DEBUG: 2022-08-22 09:44:42 : satpy.readers.satpy_cf_nc] Getting data for: I05
[DEBUG: 2022-08-22 09:44:42 : satpy.scene] Unloading dataset: DataID(name='I02', wavelength=WavelengthRange(min=0.845, central=0.865, max=0.884, unit='µm'), resolution=371, calibration=<calibration.reflectance>, modifiers=('sunz_corrected_iband',))
[DEBUG: 2022-08-22 09:44:42 : satpy.scene] Unloading dataset: DataID(name='I01', wavelength=WavelengthRange(min=0.6, central=0.64, max=0.68, unit='µm'), resolution=371, calibration=<calibration.reflectance>, modifiers=('sunz_corrected_iband',))
[DEBUG: 2022-08-22 09:44:42 : satpy.scene] Unloading dataset: DataID(name='I05', wavelength=WavelengthRange(min=10.5, central=11.45, max=12.3, unit='µm'), resolution=371, calibration=<calibration.brightness_temperature>, modifiers=())
[DEBUG: 2022-08-22 09:44:43 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-22 09:44:43 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-22 09:44:43 : satpy.scene] Resampling DataID(name='hr_overview', resolution=371)
[INFO: 2022-08-22 09:44:43 : satpy.scene] Not reducing data before resampling.
[INFO: 2022-08-22 09:44:43 : satpy.resample] Using default KDTree resampler
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/pyresample/geometry.py:528: DeprecationWarning: This function is deprecated. See: https://pyproj4.github.io/pyproj/stable/gotchas.html#upgrading-to-pyproj-2-from-pyproj-1
  xyz = np.stack(transform(src, dst, lons, lats, alt), axis=1)
[DEBUG: 2022-08-22 09:44:44 : satpy.resample] Computing kd-tree parameters
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/pyresample/kd_tree.py:1025: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  input_coords = input_coords.astype(np.float)
[DEBUG: 2022-08-22 09:44:44 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-22 09:44:44 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/pyresample/kd_tree.py:1047: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  radius=self.radius_of_influence, dtype=np.int,
[DEBUG: 2022-08-22 09:44:44 : satpy.resample] Resampling where-5fb41efbf8e102a6b14753496a2416fc
[DEBUG: 2022-08-22 09:44:44 : satpy.writers] Reading ['/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/satpy/etc/writers/simple_image.yaml']
[DEBUG: 2022-08-22 09:44:44 : satpy.writers] Enhancement configuration options: [{'name': 'inverse', 'method': <function invert at 0x7ff5b77dd280>, 'args': [[False, False, True]]}, {'name': 'stretch', 'method': <function stretch at 0x7ff5b77dddc0>, 'kwargs': {'stretch': 'linear'}}, {'name': 'gamma', 'method': <function gamma at 0x7ff5b77ddd30>, 'kwargs': {'gamma': 1.6}}]
[DEBUG: 2022-08-22 09:44:44 : trollimage.xrimage] Applying invert with parameters [False, False, True]
[DEBUG: 2022-08-22 09:44:44 : trollimage.xrimage] Applying stretch linear with parameters {}
[DEBUG: 2022-08-22 09:44:44 : trollimage.xrimage] Perform a linear contrast stretch.
[DEBUG: 2022-08-22 09:44:44 : trollimage.xrimage] Calculate the histogram quantiles: 
[DEBUG: 2022-08-22 09:44:44 : trollimage.xrimage] Left and right quantiles: 0.005 0.005
[DEBUG: 2022-08-22 09:44:44 : trollimage.xrimage] Applying gamma 1.6
[DEBUG: 2022-08-22 09:44:44 : satpy.writers.simple_image] Saving to image: hr_overview-20220818-102458.png
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing BlpImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing BmpImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing BufrStubImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing CurImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing DcxImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing DdsImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing EpsImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing FitsStubImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing FliImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing FpxImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing FtexImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing GbrImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing GifImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing GribStubImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing Hdf5StubImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing IcnsImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing IcoImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing ImImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing ImtImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing IptcImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing JpegImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing Jpeg2KImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing McIdasImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing MicImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing MpegImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing MpoImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing MspImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing PalmImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing PcdImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing PcxImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing PdfImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing PixarImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing PngImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing PpmImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing PsdImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing SgiImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing SpiderImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing SunImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing TgaImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing TiffImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing WebPImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing WmfImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing XbmImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing XpmImagePlugin
[DEBUG: 2022-08-22 09:44:44 : PIL.Image] Importing XVThumbImagePlugin
[DEBUG: 2022-08-22 09:44:44 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-22 09:45:47 : trollimage.xrimage] Interval: left=<xarray.DataArray (bands: 3)>
array([   3.28414226,    1.47481191, -301.93130341])
Coordinates:
    quantile  float64 0.005
Dimensions without coordinates: bands, right=<xarray.DataArray (bands: 3)>
array([  87.22452698,   98.27553253, -224.84414673])
Coordinates:
    quantile  float64 0.995
Dimensions without coordinates: bands

real	1m7.671s
user	0m26.166s
sys	0m5.934s

0.37.0

time python test-thredds-satpy-netcdf.py /lustre/storeB/immutable/archive/projects/remotesensing/satellite-thredds/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc 
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
[DEBUG: 2022-08-22 10:11:56 : satpy.readers.yaml_reader] Reading ('/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/satpy/etc/readers/satpy_cf_nc.yaml',)
[DEBUG: 2022-08-22 10:11:56 : satpy.readers.yaml_reader] Assigning to satpy_cf_nc: ['/lustre/storeB/immutable/archive/projects/remotesensing/satellite-thredds/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc']
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/cfgrib/xarray_plugin.py:10: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(xr.__version__) <= "0.17.0":
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/satpy/readers/satpy_cf_nc.py:240: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.
  if 'modifiers' in ds_info and not ds_info['modifiers']:
['I01', 'I02', 'I03', 'I04', 'I05', 'latitude', 'longitude', 'satellite_azimuth_angle', 'satellite_zenith_angle', 'solar_azimuth_angle', 'solar_zenith_angle']
[DEBUG: 2022-08-22 10:11:57 : satpy.composites.config_loader] Looking for composites config file viirs.yaml
[DEBUG: 2022-08-22 10:11:57 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2022-08-22 10:11:57 : satpy.node] Skipping optional DataQuery(wavelength=13.4, calibration='brightness_temperature'): Unknown dataset DataQuery(wavelength=13.4, calibration='brightness_temperature')
['cloudtop', 'green_snow', 'hr_cloudtop_daytime', 'hr_overview', 'ir108_3d', 'ir_cloud_day', 'natural_color_raw', 'natural_color_sun', 'snow', 'snow_hires', 'ssec_fog']
[DEBUG: 2022-08-22 10:11:57 : satpy.readers.satpy_cf_nc] Getting data for: I02
[DEBUG: 2022-08-22 10:11:57 : satpy.readers.satpy_cf_nc] Getting data for: I05
[DEBUG: 2022-08-22 10:11:57 : satpy.readers.satpy_cf_nc] Getting data for: I01
[DEBUG: 2022-08-22 10:11:58 : satpy.scene] Unloading dataset: DataID(name='I02', wavelength=WavelengthRange(min=0.845, central=0.865, max=0.884, unit='µm'), resolution=371, calibration=<calibration.reflectance>, modifiers=('sunz_corrected_iband',))
[DEBUG: 2022-08-22 10:11:58 : satpy.scene] Unloading dataset: DataID(name='I05', wavelength=WavelengthRange(min=10.5, central=11.45, max=12.3, unit='µm'), resolution=371, calibration=<calibration.brightness_temperature>, modifiers=())
[DEBUG: 2022-08-22 10:11:58 : satpy.scene] Unloading dataset: DataID(name='I01', wavelength=WavelengthRange(min=0.6, central=0.64, max=0.68, unit='µm'), resolution=371, calibration=<calibration.reflectance>, modifiers=('sunz_corrected_iband',))
[DEBUG: 2022-08-22 10:11:58 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-22 10:11:58 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-22 10:11:58 : satpy.scene] Resampling DataID(name='hr_overview', resolution=371)
[INFO: 2022-08-22 10:11:58 : satpy.scene] Not reducing data before resampling.
[INFO: 2022-08-22 10:11:58 : satpy.resample] Using default KDTree resampler
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/pyresample/geometry.py:528: DeprecationWarning: This function is deprecated. See: https://pyproj4.github.io/pyproj/stable/gotchas.html#upgrading-to-pyproj-2-from-pyproj-1
  xyz = np.stack(transform(src, dst, lons, lats, alt), axis=1)
[DEBUG: 2022-08-22 10:11:59 : satpy.resample] Computing kd-tree parameters
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/pyresample/kd_tree.py:1025: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  input_coords = input_coords.astype(np.float)
[DEBUG: 2022-08-22 10:11:59 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-22 10:11:59 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/pyresample/kd_tree.py:1047: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  radius=self.radius_of_influence, dtype=np.int,
[DEBUG: 2022-08-22 10:11:59 : satpy.resample] Resampling where-5fb41efbf8e102a6b14753496a2416fc
[DEBUG: 2022-08-22 10:11:59 : satpy.writers] Reading ['/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/satpy/etc/writers/simple_image.yaml']
[DEBUG: 2022-08-22 10:11:59 : satpy.writers] Enhancement configuration options: [{'name': 'inverse', 'method': <function invert at 0x7f59fd53ec10>, 'args': [[False, False, True]]}, {'name': 'stretch', 'method': <function stretch at 0x7f59fd53e3a0>, 'kwargs': {'stretch': 'linear'}}, {'name': 'gamma', 'method': <function gamma at 0x7f59fd53e0d0>, 'kwargs': {'gamma': 1.6}}]
[DEBUG: 2022-08-22 10:11:59 : trollimage.xrimage] Applying invert with parameters [False, False, True]
[DEBUG: 2022-08-22 10:11:59 : trollimage.xrimage] Applying stretch linear with parameters {}
[DEBUG: 2022-08-22 10:11:59 : trollimage.xrimage] Perform a linear contrast stretch.
[DEBUG: 2022-08-22 10:11:59 : trollimage.xrimage] Calculate the histogram quantiles: 
[DEBUG: 2022-08-22 10:11:59 : trollimage.xrimage] Left and right quantiles: 0.005 0.005
[DEBUG: 2022-08-22 10:11:59 : trollimage.xrimage] Applying gamma 1.6
[DEBUG: 2022-08-22 10:11:59 : satpy.writers.simple_image] Saving to image: hr_overview-20220818-102458.png
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing BlpImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing BmpImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing BufrStubImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing CurImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing DcxImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing DdsImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing EpsImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing FitsStubImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing FliImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing FpxImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing FtexImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing GbrImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing GifImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing GribStubImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing Hdf5StubImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing IcnsImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing IcoImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing ImImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing ImtImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing IptcImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing JpegImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing Jpeg2KImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing McIdasImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing MicImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing MpegImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing MpoImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing MspImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing PalmImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing PcdImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing PcxImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing PdfImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing PixarImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing PngImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing PpmImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing PsdImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing SgiImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing SpiderImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing SunImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing TgaImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing TiffImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing WebPImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing WmfImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing XbmImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing XpmImagePlugin
[DEBUG: 2022-08-22 10:11:59 : PIL.Image] Importing XVThumbImagePlugin
[DEBUG: 2022-08-22 10:11:59 : pyproj] PROJ_ERROR: proj_crs_get_sub_crs: Object is not a CompoundCRS
[DEBUG: 2022-08-22 10:13:02 : trollimage.xrimage] Interval: left=<xarray.DataArray (bands: 3)>
array([   3.28414226,    1.47481191, -301.93130341])
Coordinates:
    quantile  float64 0.005
Dimensions without coordinates: bands, right=<xarray.DataArray (bands: 3)>
array([  87.22452698,   98.27553253, -224.84414673])
Coordinates:
    quantile  float64 0.995
Dimensions without coordinates: bands

real	1m8.194s
user	0m24.117s
sys	0m5.900s

So: Reading from file ( over nfs mount) gives approx same timing.

Next try to remove part of the script to see where to versions differ.

TAlonglong avatar Aug 22 '22 08:08 TAlonglong

Looking 0.37.0

Stripping down the script to just load I01

import sys
from satpy import Scene
from satpy.utils import debug_on
debug_on()

file_in = sys.argv[1]

global_scene = Scene(reader='satpy_cf_nc', filenames=[file_in])

#global_scene.load(['hr_overview'])
global_scene.load(['I01'])

This is interresting: 0.37.0 loading only I01 takes just 2 seconds

time python test-thredds-satpy-netcdf.py https://thredds.met.no/thredds/dodsC/remotesensingsatellite/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
[DEBUG: 2022-08-22 10:33:15 : asyncio] Using selector: EpollSelector
[DEBUG: 2022-08-22 10:33:15 : satpy.readers.yaml_reader] Reading ('/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/satpy/etc/readers/satpy_cf_nc.yaml',)
[DEBUG: 2022-08-22 10:33:15 : satpy.readers.yaml_reader] Assigning to satpy_cf_nc: [<FSFile "https://thredds.met.no/thredds/dodsC/remotesensingsatellite/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc">]
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/cfgrib/xarray_plugin.py:10: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(xr.__version__) <= "0.17.0":
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
[DEBUG: 2022-08-22 10:33:15 : satpy.composites.config_loader] Looking for composites config file viirs.yaml
[DEBUG: 2022-08-22 10:33:16 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2022-08-22 10:33:16 : satpy.readers.satpy_cf_nc] Getting data for: I01

real	0m1.847s
user	0m1.896s
sys	0m1.692s

while loading the hr_overview composite takes 4 minutes:

time python test-thredds-satpy-netcdf.py https://thredds.met.no/thredds/dodsC/remotesensingsatellite/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
[DEBUG: 2022-08-22 10:20:02 : asyncio] Using selector: EpollSelector
[DEBUG: 2022-08-22 10:20:02 : satpy.readers.yaml_reader] Reading ('/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/satpy/etc/readers/satpy_cf_nc.yaml',)
[DEBUG: 2022-08-22 10:20:02 : satpy.readers.yaml_reader] Assigning to satpy_cf_nc: [<FSFile "https://thredds.met.no/thredds/dodsC/remotesensingsatellite/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc">]
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/cfgrib/xarray_plugin.py:10: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(xr.__version__) <= "0.17.0":
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
['I01', 'I02', 'I03', 'I04', 'I05', 'latitude', 'longitude', 'satellite_azimuth_angle', 'satellite_zenith_angle', 'solar_azimuth_angle', 'solar_zenith_angle']
[DEBUG: 2022-08-22 10:20:02 : satpy.composites.config_loader] Looking for composites config file viirs.yaml
[DEBUG: 2022-08-22 10:20:02 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2022-08-22 10:20:02 : satpy.node] Skipping optional DataQuery(wavelength=13.4, calibration='brightness_temperature'): Unknown dataset DataQuery(wavelength=13.4, calibration='brightness_temperature')
['cloudtop', 'green_snow', 'hr_cloudtop_daytime', 'hr_overview', 'ir108_3d', 'ir_cloud_day', 'natural_color_raw', 'natural_color_sun', 'snow', 'snow_hires', 'ssec_fog']
[DEBUG: 2022-08-22 10:20:02 : satpy.readers.satpy_cf_nc] Getting data for: I05
[DEBUG: 2022-08-22 10:20:02 : satpy.readers.satpy_cf_nc] Getting data for: I02
[DEBUG: 2022-08-22 10:20:02 : satpy.readers.satpy_cf_nc] Getting data for: I01
[DEBUG: 2022-08-22 10:24:15 : satpy.scene] Unloading dataset: DataID(name='I05', wavelength=WavelengthRange(min=10.5, central=11.45, max=12.3, unit='µm'), resolution=371, calibration=<calibration.brightness_temperature>, modifiers=())
[DEBUG: 2022-08-22 10:24:15 : satpy.scene] Unloading dataset: DataID(name='I02', wavelength=WavelengthRange(min=0.845, central=0.865, max=0.884, unit='µm'), resolution=371, calibration=<calibration.reflectance>, modifiers=('sunz_corrected_iband',))
[DEBUG: 2022-08-22 10:24:15 : satpy.scene] Unloading dataset: DataID(name='I01', wavelength=WavelengthRange(min=0.6, central=0.64, max=0.68, unit='µm'), resolution=371, calibration=<calibration.reflectance>, modifiers=('sunz_corrected_iband',))

real	4m14.290s
user	0m52.090s
sys	0m23.374s

TAlonglong avatar Aug 22 '22 08:08 TAlonglong

For 0.36.0 loading hr_overview takes less than 2 seconds

time python test-thredds-satpy-netcdf.py https://thredds.met.no/thredds/dodsC/remotesensingsatellite/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
[DEBUG: 2022-08-22 10:48:51 : satpy.readers.yaml_reader] Reading ('/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/satpy/etc/readers/satpy_cf_nc.yaml',)
[DEBUG: 2022-08-22 10:48:51 : satpy.readers.yaml_reader] Assigning to satpy_cf_nc: ['https://thredds.met.no/thredds/dodsC/remotesensingsatellite/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc']
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/cfgrib/xarray_plugin.py:10: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(xr.__version__) <= "0.17.0":
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
[DEBUG: 2022-08-22 10:48:51 : satpy.composites.config_loader] Looking for composites config file viirs.yaml
[DEBUG: 2022-08-22 10:48:51 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2022-08-22 10:48:51 : satpy.readers.satpy_cf_nc] Getting data for: I01
[DEBUG: 2022-08-22 10:48:51 : satpy.readers.satpy_cf_nc] Getting data for: I02
[DEBUG: 2022-08-22 10:48:51 : satpy.readers.satpy_cf_nc] Getting data for: I05
[DEBUG: 2022-08-22 10:48:51 : satpy.scene] Unloading dataset: DataID(name='I01', wavelength=WavelengthRange(min=0.6, central=0.64, max=0.68, unit='µm'), resolution=371, calibration=<calibration.reflectance>, modifiers=('sunz_corrected_iband',))
[DEBUG: 2022-08-22 10:48:51 : satpy.scene] Unloading dataset: DataID(name='I02', wavelength=WavelengthRange(min=0.845, central=0.865, max=0.884, unit='µm'), resolution=371, calibration=<calibration.reflectance>, modifiers=('sunz_corrected_iband',))
[DEBUG: 2022-08-22 10:48:51 : satpy.scene] Unloading dataset: DataID(name='I05', wavelength=WavelengthRange(min=10.5, central=11.45, max=12.3, unit='µm'), resolution=371, calibration=<calibration.brightness_temperature>, modifiers=())

real	0m1.900s
user	0m1.891s
sys	0m1.694s

And loading only I01 takes also less than 2 seconds:

time python test-thredds-satpy-netcdf.py https://thredds.met.no/thredds/dodsC/remotesensingsatellite/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
[DEBUG: 2022-08-22 10:50:50 : satpy.readers.yaml_reader] Reading ('/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/satpy/etc/readers/satpy_cf_nc.yaml',)
[DEBUG: 2022-08-22 10:50:50 : satpy.readers.yaml_reader] Assigning to satpy_cf_nc: ['https://thredds.met.no/thredds/dodsC/remotesensingsatellite/polar-swath/2022/08/18/npp-viirs-iband-20220818102458-20220818103910.nc']
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/cfgrib/xarray_plugin.py:10: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(xr.__version__) <= "0.17.0":
/home/trygveas/miniconda3/envs/python/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
[DEBUG: 2022-08-22 10:50:51 : satpy.composites.config_loader] Looking for composites config file viirs.yaml
[DEBUG: 2022-08-22 10:50:51 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2022-08-22 10:50:51 : satpy.readers.satpy_cf_nc] Getting data for: I01

real	0m1.795s
user	0m1.862s
sys	0m1.624s

TAlonglong avatar Aug 22 '22 08:08 TAlonglong

The only difference I can spot is that 0.37.0 when assigning to satpy_cf_nc is using FSFile and then I'm guessing 0.36.0 uses some native when loading the file. Not sure if this did something https://github.com/pytroll/satpy/pull/2096

TAlonglong avatar Aug 22 '22 08:08 TAlonglong

Just to be clear, in your trimmed down script, you are still resampling and saving? Or are you calling .compute() on that result?

djhoese avatar Aug 22 '22 13:08 djhoese

If you look at my trimmed down script you see its, no, no and no.

TAlonglong avatar Aug 22 '22 13:08 TAlonglong

Wow, so that is purely just the reader. Ok, ok. We may need to bring @pnuu into this who might have an idea. My guess is this is causing xarray to use h5netcdf and streaming the bytes directly to the library whereas before OpenDAP inside the NetCDF C library was getting the http URL directly and was able to intelligently parse it.

I'm not sure how to have both #2096 and OpenDAP URLs at the same time unless we don't parse/handle http(s) URLs "magically" at all.

djhoese avatar Aug 22 '22 13:08 djhoese

So I did some digging in the code and the decision of filetype is done here I think: https://github.com/pytroll/satpy/blob/main/satpy/utils.py#L634-L647

with tests here https://github.com/pytroll/satpy/blob/main/satpy/tests/test_utils.py#L481-L569 So maybe test with http and https schema should be added. But I'm not sure of http(s) should sort under local or remote files.

TAlonglong avatar Aug 23 '22 05:08 TAlonglong

I understand what you want to achieve, however it's a minority of readers which can read directly from http/https. I would recommend instead a little mechanism in the netcdf readers to check what protocol the fsfiles represent, and pass the link directly if it turns out to be http/https.

mraspaud avatar Aug 23 '22 06:08 mraspaud

like here: https://github.com/pytroll/satpy/blob/main/satpy/readers/satpy_cf_nc.py#L287

mraspaud avatar Aug 23 '22 06:08 mraspaud

So something like this in the satpy_cf_nc reader in get_dataset just before the call to xr.open_dataset

if isinstance(self.filename, FSFile) and urlparse(str(self.filename)).scheme in ['http', 'https']:
        self.filename = str(self.filename)

It looks very ugly.

Unfortunately I'm not able to repeat the test, or I can run the test, but I don't see any difference any longer. SO I don't know what happened.

TAlonglong avatar Aug 23 '22 08:08 TAlonglong

when you tried you were away, right? could it be that you are too close to the data source now? I'll test from here, see if I can see it.

mraspaud avatar Aug 23 '22 12:08 mraspaud

Just loading runs in about 3 seconds from my computer, independently of the version (0.36.0, 0.37.0, main)...

mraspaud avatar Aug 23 '22 12:08 mraspaud

Same here. Just did mamba update --all and updated to latest Satpy main and the whole example script (the I01 variant) takes only 3 seconds to run.

pnuu avatar Aug 23 '22 13:08 pnuu