integrate notebooks

[egouden]

I see several reasons to integrate the notebooks in the main wradlib repository:

• integrate notebook in testing
• avoid breaking change
• better visibility for users
• follow xradar strategy

Update by @kmuehlbauer : This here is the current layout. Some few files are added twice into the tree, because they might be relevant to those sections.

As agreed we tackle this one by one or group by group. In #714 I've fixed the rendering pipeline and kept one simple plot notebook for starters. I've secured the complete markdown only notebooks here https://github.com/wradlib/wradlib/tree/8c2ae6f341d39afd724aaa382e6e8d542c90e226/examples/notebooks, so we can build upon them.

• [ ] Python Intro

  • [ ] A quick start to Python
  • [ ] Numpy: manipulating numerical data
  • [ ] Visualising data with Matplotlib
  • [ ] Dealing with time series

• [ ] wradlib Basics

  • [ ] Get wradlib version
  • [ ] Typical Workflow for Radar-Based Rainfall Estimation
  • [ ] From Reflectivity to Rainfall
  • [ ] wradlib in an hour

• [ ] Data Input/Output

  • [ ] Legacy readers

    • [ ] DWD DX
    • [ ] NetCDF
    • [ ] HDF5
    • [ ] OPERA ODIM
    • [ ] GAMIC HDF
    • [ ] Leonardo Rainbow
    • [ ] Vaisala IRIS/Sigmet

  • [ ] Xarray readers

    • [ ] RADOLAN
    • [ ] ODIM
    • [ ] GAMIC
    • [ ] CfRadial1
    • [ ] CfRadial2
    • [ ] Iris/Sigmet
    • [ ] Rainbow5
    • [ ] Furuno SCN/SCNX

  • [ ] DWD RADOLAN

    • [ ] RADOLAN Quick Start
    • [ ] RADOLAN Binary Data Format
    • [ ] RADOLAN Product Showcase
    • [ ] RADOLAN Grid
    • [ ] DWD Radar Network

  • [ ] GIS Vector Data

  • [ ] GIS Raster Export

• [ ] Visualization

  • [ ] Plot PPI
  • [ ] Plot RHI
  • [ ] Plot Curvelinear Grids
  • [ ] Plot geodata
  • [ ] Plot geodata with cartopy
  • [ ] Plot radar scan strategy

• [ ] Attenuation Correction

• [ ] Beam Blockage

• [ ] Echo Classification

  • [ ] Clutter detection using Gabella approach
  • [ ] Clutter detection by using space-born cloud images
  • [ ] Heuristic clutter detection
  • [ ] Fuzzy echo classification
  • [ ] 2D Membershipfunction HMC
  • [ ] Hydrometeor partitioning ratio - GPM
  • [ ] Hydrometeor partitioning ratio - GroundRadar

• [ ] Georeferencing

  • [ ] Computing Cartesian Coordinates from Polar Data
  • [ ] Georeferencing a Radar Dataset
  • [ ] RADOLAN Grid Calculations
  • [ ] Overlay Ancillary Data
  • [ ] Managing Georeferenced Data

• [ ] Interpolation

• [ ] Compositing

  • [ ] Production of a maximum reflectivity composite
  • [ ] Comparison of transformation methods

• [ ] Gauge Adjustment

• [ ] Verification

• [ ] Zonal Statistics

  • [ ] Quickstart
  • [ ] Cartesian Grid
  • [ ] Polar Grid

• [ ] Recipes

  • [ ] Recipe1: Clutter and attenuation correction plus composition for two DWD radars
  • [ ] Recipe2: Reading and visualizing an ODIM_H5 polar volume
  • [ ] Recipe3: Match spaceborn SR (GPM/TRRM) with ground radars GR
  • [ ] Recipe4: Load ODIM_H5 Volume data from DWD Open Data
  • [ ] Recipe5: Zonalstats on Cartesian Grid
  • [ ] Recipe6: Zonalstats on Polar Grid

[kmuehlbauer]

Thanks Edouard for creating this issue.

I agree. The root causes to have a dedicated notebooks-repo do not exist any more and the handling now is a bit of a pain. We have brought the docs back already in v2.0.0 for the same reason and I think it's time to also include the notebooks.

This might look like an easy task, but we should discuss this in more detail.

What I would like bring into the discussion:

• use [MyST]](https://mystmd.org/) markdown for writing notebooks instead of regular notebooks
• drop Python intro, instead link to other resources (NumPy, SciPy etc)
• restructure the tutorials (eg. dedicated Getting started section with low barrier simple examples)
• drop backend specific notebooks which are already handled in xradar
• add a gallery reel

[egouden]

This sounds like interesting improvements to the notebooks. What is really necessary before we can migrate?

[kmuehlbauer]

The main point is that our notebook creation is quite heavy. We should not stress ReadTheDocs with that burden. Currently on GitHub CI we:

1. pull the unrendered notebooks from wradlib-notebooks
2. render them and push results to dedicated branch of wradlib-notebooks
3. trigger rtd
4. on rtd we fetch the rendered notebooks from wradlib-notebooks and use them for compiling docs

If we've moved the notebooks back into wradlib main repo, we would have to do something like this:

1. render the notebooks and push unrendered and rendered to wradlib-notebooks
2. trigger rtd
3. on rtd we fetch the rendered notebooks from wradlib-notebooks and use them for compiling docs

We could start with a simple notebook which we include between point 1. and 2. of the first block above.

[egouden]

What about:

1. disconnect the current notebook pipeline
2. build a new pipeline with selected content

[kmuehlbauer]

Can you give a bit more detail how you envision the pipeline where the notebook rendering does not run on ReadTheDocs.

The current pipeline works quite well and if we have the new notebooks in wradlib main, we can still use it. All source files in one place, only the rendered output over at wradlib-notebooks.

I can add a PR with my suggestion later this week

[egouden]

I mean to stop any work on the existing notebooks. Just keep a link to this version in the docs and label it as old. Then we work on a clean new version of the notebooks step by step by PR. After some time when the new version is better we remove the old version.

[kmuehlbauer]

Yes, sure, I get it. But that doesn't contradict with my suggestion. So all good.

I mean to stop any work on the existing notebooks. Just keep a link to this version in the docs and label it as old. Then we work on a clean new version of the notebooks step by step by PR. After some time when the new version is better we remove the old version.

We need to be careful with renaming, to keep search engines happy.

[egouden]

Looking forward your suggestion then !

[kmuehlbauer]

@egouden I'm on it, but this will take a bit more time as expected. I'm following multiple path's now, and hope to have a PR ready early next week. Thanks for your patience.

[kmuehlbauer]

Update: With #731 we now have the following situation:

• RADOLAN Guide is now moved into it's own repo

  • Repo: https://github.com/wradlib/radolan-guide
  • Docs: https://docs.wradlib.org/projects/radolan

• wradlib-notebooks are reractored to myst-notebooks and the docs are created directly in the repo

  • Repo: https://github.com/wradlib/wradlib-notebooks
  • Docs: https://docs.wradlib.org/projects/notebooks

New myst notebooks, which cover core wradlib features are now directly inside docs/notebooks

wradlib-notebooks (see above) are still reachable via "Tutorials and Examples" Link from wradlib documentation https://docs.wradlib.org/en/latest/.