Add Scene function to use Hvplot backend visualization

[bornagain1981]

This pull request is my attempt to do something similar to "to_geoviews" function. Hvplot syntax is easier than geoviews. In this case input datasets are optional. If no dataset is selected as input all scene variable keys is plotted and available in the output as obj.variable It is no very fast, especially when there are many datasets and/or high definition coastlines. But for the time being it is ok. Future improvements may be to use multiprocessing library inside the function.

[bornagain1981]

I added hvplot in extra and test sections of setup.py . But hvplot not found error keeps happening

[djhoese]

Add it to https://github.com/pytroll/satpy/blob/main/continuous_integration/environment.yaml

Also, please wrap the import in scene.py in a try/except like:

try:
    import hvplot
except ImportError:
    hvplot = None

then later in your code do something like if hvplot is None: raise ImportError("'hvplot' must be installed to use this feature")

[codecov[bot]]

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (56df5da) 95.35% compared to head (9b0bcae) 95.35%.

Files	Patch %	Lines
satpy/scene.py	90.90%	3 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2106      +/-   ##
==========================================
- Coverage   95.35%   95.35%   -0.01%     
==========================================
  Files         371      371              
  Lines       52464    52521      +57     
==========================================
+ Hits        50028    50082      +54     
- Misses       2436     2439       +3     

Flag	Coverage Δ
behaviourtests	4.15% <3.44%> (-0.01%)	:arrow_down:
unittests	95.97% <94.82%> (-0.01%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[coveralls]

Coverage: 95.264% (-0.06%) from 95.329% when pulling 09ad15d414e40a3dbcca4347bac647149a6071b3 on bornagain1981:main into b92fe9ebb62c30e60f4d7991742fc9a3e5236a77 on pytroll:main.

[gerritholl]

What is the status of this PR? Any chance of resolving the merge conflicts and looking at the remaining reviewer comments?

[bornagain1981]

What is the status of this PR? Any chance of resolving the merge conflicts and looking at the remaining reviewer comments?

yes, I hope to be able to solve them in these vacation days

[djhoese]

It looks like there are still merge conflicts (you may need to update your fork's main branch with upstream satpy main). Also, could you add some tests for this?

[mraspaud]

@bornagain1981 any news on this?

[bornagain1981]

I started working on my PR again. I merged the current version of satpy and it’s aligned. Currently there is an error on: holoviews -> datashader - > numba

_SystemError: initialization of internal failed without raising an exception

there are issues open because it is incompatible with the numpy version. pip install numpy==1.23.0 fix.

Is it a problem to downgrade numpy?

[djhoese]

Is it a problem to downgrade numpy?

We won't be downgrading numpy for all of Satpy, but our CI is currently passing for the released versions of numpy and holoviews. Can you link to the issues you've discovered?

I know that the unstable environment in the CI is failing due to an incompatibility with holoviews and the next version of numpy (still unreleased) but this is already fixed in holoviews's github.

If you want to downgrade numpy locally because of some issue you're running into then that is up to you.

[bornagain1981]

Is it a problem to downgrade numpy?

We won't be downgrading numpy for all of Satpy, but our CI is currently passing for the released versions of numpy and holoviews. Can you link to the issues you've discovered?

---

    SystemError                               Traceback (most recent call last)
    Cell In[8], line 1
    ----> 1 sss = scn.to_hvplot(coastline='50m')
    
    File ~/git/satpy/satpy/scene.py:1113, in Scene.to_hvplot(self, datasets, *args, **kwargs)
       1111         plot[element] = _plot_rgb(xarray_ds, element, **defaults)
       1112     else:
    -> 1113         plot[element] = _plot_quadmesh(xarray_ds, element, **defaults)
       1115 return plot
    
    File ~/git/satpy/satpy/scene.py:1086, in Scene.to_hvplot.<locals>._plot_quadmesh(xarray_ds, variable, **defaults)
       1085 def _plot_quadmesh(xarray_ds, variable, **defaults):
    -> 1086     return xarray_ds[variable].hvplot.quadmesh(
       1087         clabel=f'[{_get_units(xarray_ds,variable)}]', title=title,
       1088         **defaults)
    
    File /media/bornagain/907b3896-8bf1-4f1b-bcc9-09a4c62755cf/conda_envs/test_satpy/lib/python3.9/site-packages/hvplot/plotting/core.py:2067, in hvPlot.quadmesh(self, x, y, z, colorbar, **kwds)
       2018 def quadmesh(self, x=None, y=None, z=None, colorbar=True, **kwds):
       2019     """
       2020     QuadMesh plot
       2021 
       (...)
       2065     - HoloViews: https://holoviews.org/reference/elements/bokeh/QuadMesh.html
       2066     """
    -> 2067     return self(x, y, z=z, kind="quadmesh", colorbar=colorbar, **kwds)
    
    File /media/bornagain/907b3896-8bf1-4f1b-bcc9-09a4c62755cf/conda_envs/test_satpy/lib/python3.9/site-packages/hvplot/plotting/core.py:92, in hvPlotBase.__call__(self, x, y, kind, **kwds)
         89         plot = self._get_converter(x, y, kind, **kwds)(kind, x, y)
         90         return pn.panel(plot, **panel_dict)
    ---> 92 return self._get_converter(x, y, kind, **kwds)(kind, x, y)
    
    File /media/bornagain/907b3896-8bf1-4f1b-bcc9-09a4c62755cf/conda_envs/test_satpy/lib/python3.9/site-packages/hvplot/converter.py:1249, in HoloViewsConverter.__call__(self, kind, x, y)
       1247     return layers
       1248 try:
    -> 1249     from holoviews.operation.datashader import datashade, rasterize, dynspread
       1250     from datashader import reductions
       1251 except ImportError:
    
    File /media/bornagain/907b3896-8bf1-4f1b-bcc9-09a4c62755cf/conda_envs/test_satpy/lib/python3.9/site-packages/holoviews/operation/datashader.py:10
          8 import pandas as pd
          9 import xarray as xr
    ---> 10 import datashader as ds
         11 import datashader.reductions as rd
         12 import datashader.transfer_functions as tf
    
    File /media/bornagain/907b3896-8bf1-4f1b-bcc9-09a4c62755cf/conda_envs/test_satpy/lib/python3.9/site-packages/datashader/__init__.py:8
          5 import param
          6 __version__ = str(param.version.Version(fpath=__file__, archive_commit="$Format:%h$",reponame="datashader"))
    ----> 8 from .core import Canvas                                 # noqa (API import)
          9 from .reductions import *                                # noqa (API import)
         10 from .glyphs import Point                                # noqa (API import)
    
    File /media/bornagain/907b3896-8bf1-4f1b-bcc9-09a4c62755cf/conda_envs/test_satpy/lib/python3.9/site-packages/datashader/core.py:13
         10 from xarray import DataArray, Dataset
         11 from collections import OrderedDict
    ---> 13 from .utils import Dispatcher, ngjit, calc_res, calc_bbox, orient_array, \
         14     dshape_from_xarray_dataset
         15 from .utils import get_indices, dshape_from_pandas, dshape_from_dask
         16 from .utils import Expr # noqa (API import)
    
    File /media/bornagain/907b3896-8bf1-4f1b-bcc9-09a4c62755cf/conda_envs/test_satpy/lib/python3.9/site-packages/datashader/utils.py:8
          4 import re
          6 from inspect import getmro
    ----> 8 import numba as nb
          9 import numpy as np
         10 import pandas as pd
    
    File /media/bornagain/907b3896-8bf1-4f1b-bcc9-09a4c62755cf/conda_envs/test_satpy/lib/python3.9/site-packages/numba/__init__.py:43
         39 from numba.core.decorators import (cfunc, generated_jit, jit, njit, stencil,
         40                                    jit_module)
         42 # Re-export vectorize decorators and the thread layer querying function
    ---> 43 from numba.np.ufunc import (vectorize, guvectorize, threading_layer,
         44                             get_num_threads, set_num_threads)
         46 # Re-export Numpy helpers
         47 from numba.np.numpy_support import carray, farray, from_dtype
    
    File /media/bornagain/907b3896-8bf1-4f1b-bcc9-09a4c62755cf/conda_envs/test_satpy/lib/python3.9/site-packages/numba/np/ufunc/__init__.py:3
          1 # -*- coding: utf-8 -*-
    ----> 3 from numba.np.ufunc.decorators import Vectorize, GUVectorize, vectorize, guvectorize
          4 from numba.np.ufunc._internal import PyUFunc_None, PyUFunc_Zero, PyUFunc_One
          5 from numba.np.ufunc import _internal, array_exprs
    
    File /media/bornagain/907b3896-8bf1-4f1b-bcc9-09a4c62755cf/conda_envs/test_satpy/lib/python3.9/site-packages/numba/np/ufunc/decorators.py:3
          1 import inspect
    ----> 3 from numba.np.ufunc import _internal
          4 from numba.np.ufunc.parallel import ParallelUFuncBuilder, ParallelGUFuncBuilder
          6 from numba.core.registry import TargetRegistry
    
    SystemError: initialization of _internal failed without raising an exception

[djhoese]

Ok looks like this is purely a numba incompatibility with numpy 1.24.0, not holoviews. Looks like a PR to support numpy 1.24 was just merged a couple weeks ago: https://github.com/numba/numba/pull/8691

It has not been released yet and I don't see a milestone in numba that is anywhere near release: https://github.com/numba/numba/milestones

Downgrading numpy locally for development may be your only option. If you installed everything with conda-forge I would assume the version limits should "just work" and you should get a working environment.

[mraspaud]

Looks good, but would there be a way to add a unittest?

[coveralls]

Pull Request Test Coverage Report for Build 7220473626

• 55 of 58 (94.83%) changed or added relevant lines in 2 files are covered.
• No unchanged relevant lines lost coverage.
• Overall first build on main at 95.919%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
satpy/scene.py	30	33	90.91%
<!--	Total:	55	58

Totals
Change from base Build 7220123684:	95.9%
Covered Lines:	50208
Relevant Lines:	52344

---

💛 - Coveralls

[bornagain1981]

I added the holoviews module in setup file but Sphinx keep reporting "missing module error"

[bornagain1981]

Couple more requests after running the code. And the tests :-)

I added the tests :)

[djhoese]

Could we please get a follow up pull request that moves this functionality to a function in:

https://github.com/pytroll/satpy/blob/main/satpy/_scene_converters.py

This is a lot of code to add to an already bloated scene.py module. This _scene_converters.py module was created (admittedly after this PR was started but before it was merged) as a "dumping ground" for these types of conversions.