Make it easy to use dataclass like models using familiar apis

[MarcSkovMadsen]

Superseedes #6892.

Also motivated by me trying to demonstrate that you can just as well use Panel for geospatial applications as Solara by creating apps similar to https://github.com/opengeos/solara-geospatial/tree/main/pages. But currently Panel is harder to use because it requires adding more code for using observer pattern.

Scope: Currently ipywidgets, Pydantic models

Easy to view docs

• How-To ipywidgets
• How-To pydantic
• Reference Guide ipywidgets

Todo

• [x] Create design proposal
• [x] Get initial design feedback
• [x] Update design proposal
• [x] Implement design
• [x] Clean Up
  • [x] names
  • [x] docstrings
  • [x] Add examples to docstring
  • [x] missing tests
• [x] Make functionality covers functionality needed to wrap and share ipywidget as reusable component with Viewer interface.
• [x] Support ipywidgets
• [x] Support pydantic
• [x] Document
  • [x] Update IPyWidget reference notebook
    • [x] remove example files.
  • [x] Create how-to guide for wrapping ipywidget using functionality in panel.ipywidget.
  • [x] Philipp?: Fix broken dev docs: TypeError: Cannot read properties of undefined (reading 'loader').
  • [x] Philipp?: Fix test error. I don't know how to fix the markdown docs test failure.
  • [x] Philipp?: Fix self.param.add_parameter(parameter, param.Parameter())
  • [x] Philipp?: Fix JSON pane updating before model issue.
  • [x] Build documentation and test
• [ ] Fix issues identified
  • [x] Fix not working create_parameter of ipywidgets.
  • [x] Fix missing pydantic docs dependency
  • [x] Add create_parameter for pydantic to add appropriate types of parameters.
  • [x] Fix ipywidget tuple length issue
  • [ ] Philipp?: Add sqlite as a dependency for running the ipywidgets docs in pyodide
• [ ] Test complex use cases with multiple models and sessions
• [ ] Final review and update

Maybe later

• [ ] Take inspiration from https://github.com/jmosbacher/pydantic-panel/blob/master/pydantic_panel/init.py.
  • [x] Most features supported
  • [ ] BaseModel field supported
• [ ] Take inspiration from https://github.com/LukasMasuch/streamlit-pydantic
• [ ] Take inspiration from https://pydf-docs.onrender.com/ or https://community.plotly.com/t/dash-pydantic-form/84435 for inspiration.
• [ ] Improve ModelForm.
  • [x] Support construction using instance as an alternative to class.
• [ ] Align traitlets type support with pydantic type support.

Promotion

Note: Features have been moved to panel.dataclass module since this video was made. WidgetViewer has been renamed to ModelViewer.

https://github.com/holoviz/panel/assets/42288570/83edfc2d-353d-47b8-8cd1-b0163af0a74d

Design Principles

• I've tried to create small pieces of functionality that build up to the simple to use ModelViewer class and create_rx function such that there are no dead ends and its testable.
• I've tried to use naming conventions that would be general enough to accommodate similar functionality for dataclasses, Pydantic etc. one day.
• I've not gone into creating parameters of similar types to the traits. For now that is something the developer must do if that is needed.

[MarcSkovMadsen]

Hi @philippjfr . Would you take a look at the design spec, i.e. the current files? Thanks.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 94.89144% with 40 lines in your changes missing coverage. Please review.

Project coverage is 81.94%. Comparing base (7267b38) to head (d364cfe). Report is 7 commits behind head on main.

Files	Patch %	Lines
panel/_dataclasses/base.py	86.71%	19 Missing :warning:
panel/_dataclasses/ipywidget.py	86.66%	8 Missing :warning:
panel/_dataclasses/pydantic.py	89.65%	6 Missing :warning:
panel/tests/dataclasses/test_dataclass.py	99.00%	4 Missing :warning:
panel/dataclass.py	97.39%	3 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6912      +/-   ##
==========================================
+ Coverage   81.71%   81.94%   +0.22%     
==========================================
  Files         326      331       +5     
  Lines       48082    48861     +779     
==========================================
+ Hits        39292    40040     +748     
- Misses       8790     8821      +31     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[jbednar]

Can pn.panel support auto-detecting an ipywidget so that pn.panel(leaflet_map) would work, as it's much easier to remember than pn.wrappers.ipywidget.WidgetViewer(model=leaflet_map)?

[philippjfr]

Would be a breaking change but really shouldn't break much so probably worth doing.

[MarcSkovMadsen]

The reason why its deeply nested is that there is already a panel.ipywidget module which is really panel.io.ipywidget.

And i hope the One day there will be a pydantic module with the same functionality.

Note I've renamed wrappers to observers. It relates to the observer pattern.

[MarcSkovMadsen]

pn.panel(some_ipywidget) already works as it returns the IPyWidget pane.

[MarcSkovMadsen]

Can pn.panel support auto-detecting an ipywidget so that pn.panel(leaflet_map) would work, as it's much easier to remember than pn.wrappers.ipywidget.WidgetViewer(model=leaflet_map)?

The functionality is now exposed via panel.dataclass. The idea is that the functionality/ API can accomodate dataclasses, pydantic etc. in the future.

pn.dataclass.ModelViewer(model=leaflet_map)

or

pn.dataclass.ModelParameterized(model=some_pydantic_instance)

or

pn.dataclass.create_rx(model=some_dataclass_instance, name="some_attr")

[MarcSkovMadsen]

There is not so much more I can do right now. The final thing would be to review dev docs. But its broken: TypeError: Cannot read properties of undefined (reading 'loader'). I also need help fixing the one failing test. I don't know how to structure the docs to fix the test error.

I will be waiting for dev docs fix, test fix and reviews/ feedback.

[MarcSkovMadsen]

Ok. I did one more thing. Add support for Pydantic models.

I did it because

• Its extremely useful
• It helps generalize the code to enable us to avoid breaking changes in the future.

[philippjfr]

I would like to suggest the following renames:

• create_parameterized -> to_parameterized
• create_viewer -> to_panel
• create_rx -> to_rx

Also I'm still trying to iterate on the sync method API, I would really love it if we didn't need 3 different methods there but I get why you added them.

[MarcSkovMadsen]

I would like to suggest the following renames:

• create_parameterized -> to_parameterized
• create_viewer -> to_panel
• create_rx -> to_rx

Also I'm still trying to iterate on the sync method API, I would really love it if we didn't need 3 different methods there but I get why you added them.

You can just use the sync_xyz method to create a method like the pseudo method below

def sync(model, parameterized_or_rx, names):
    if isinstance(parameterized_or_rx, param.rx) and isinstance(names, Iterable):
        sync_to_rx(model, parameterized_or_rx, *names)
    elif isinstance(parameterized_or_rx, pn.widgets.Widget) and isinstance(names, str):
        sync_to_widget(model, parameterized_or_rx, names)
    elif isinstance(parameterized_or_rx, param.Parameterized):
        sync_to_parameterized(model, parameterized_or_rx, names)
    raise ValueError()

Maybe def sync(model, parameterized_or_rx, *names_args, **names_kwargs) is nicer to use because you avoid creating the list or dictionary?

[philippjfr]

You can just use the sync_xyz method to create a method like the pseudo method below

Right, just wondering whether that's too overloaded.

[MarcSkovMadsen]

I have starting adding support for specific parameter types. Its easy. But i need to refactor the Way initial values are assigned.

[philippjfr]

I have starting adding support for specific parameter types. Its easy. But i need to refactor the Way initial values are assigned.

Ooops, missed this. I just pushed it.

[MarcSkovMadsen]

I have starting adding support for specific parameter types. Its easy. But i need to refactor the Way initial values are assigned.

Ooops, missed this. I just pushed it.

No worries. I'm happy.

[philippjfr]

Not sure I love panel.dataclass naming. While yes, they're dataclass-like things they're not actually dataclasses and I'd rather focus on the functionality this provides, i.e. the fact that this provides a compatibility layer or bridge between the libraries.

[MarcSkovMadsen]

Not sure I love panel.dataclass naming. While yes, they're dataclass-like things they're not actually dataclasses and I'd rather focus on the functionality this provides, i.e. the fact that this provides a compatibility layer or bridge between the libraries.

1. panel.compat

Explanation: This name focuses on the core functionality of the module, which is to provide compatibility between different frameworks. It conveys the idea that this module helps integrate various dataclass-like libraries with HoloViz Param.

2. panel.bridge

Explanation: The term "bridge" clearly indicates that the module serves as a connector between disparate systems. It emphasizes the role of the module in linking different frameworks, highlighting its purpose without misrepresenting the nature of the classes involved.

3. panel.integration

Explanation: This name underscores the module's role in facilitating integration between HoloViz Param and other frameworks like traitlets, ipywidgets, and pydantic. It suggests a broader scope of bringing together various components, focusing on the unifying function of the module.

Each of these names highlights the module's purpose of enhancing interoperability and integration between HoloViz Param and other dataclass-like frameworks, without suggesting that the module contains actual dataclasses.

[philippjfr]

Docs build now working.

[MarcSkovMadsen]

When I click the run with pyodide button I see

It seems we need to add sqlite as a dependency.

[MarcSkovMadsen]

FIXED

I can see that the ipywidgets create_parameter is not working for the example

import panel as pn
import ipyleaflet as ipyl

pn.extension("ipywidgets")

leaflet_map = ipyl.Map(zoom=4)

viewer = pn.dataclass.ModelViewer(model=leaflet_map, sizing_mode="stretch_both")
pn.Row(pn.Column(viewer.param, scroll=True), viewer, height=400).servable()

The problem is that the trait is an instance and not a type

If I change to use type(...)

then it can error

ValueError: Attribute 'length' of Tuple parameter 'Map.bounds' is not of the correct length (0 instead of 2).

Traceback (most recent call last):
  File "/home/jovyan/repos/private/panel/panel/_dataclasses/base.py", line 156, in sync_with_parameterized
    setattr(model, field, parameter_value)
  File "/home/jovyan/repos/private/panel/.venv/lib/python3.11/site-packages/traitlets/traitlets.py", line 715, in __set__
    raise TraitError('The "%s" trait is read-only.' % self.name)
traitlets.traitlets.TraitError: The "bounds" trait is read-only.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jovyan/repos/private/panel/panel/io/handlers.py", line 389, in run
    exec(self._code, module.__dict__)
  File "/home/jovyan/repos/private/panel/script.py", line 8, in <module>
    viewer = pn.dataclass.ModelViewer(model=leaflet_map, sizing_mode="stretch_both")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/repos/private/panel/panel/dataclass.py", line 159, in __init__
    super().__init__(**params)
  File "/home/jovyan/repos/private/panel/panel/viewable.py", line 302, in __init__
    super().__init__(**params)
  File "/home/jovyan/repos/private/panel/panel/dataclass.py", line 106, in __init__
    utils.sync_with_parameterized(self.model, self, names=names)
  File "/home/jovyan/repos/private/panel/panel/_dataclasses/base.py", line 159, in sync_with_parameterized
    setattr(parameterized, parameter, field_value)
  File "/home/jovyan/repos/private/panel/.venv/lib/python3.11/site-packages/param/parameterized.py", line 528, in _f
    instance_param.__set__(obj, val)
  File "/home/jovyan/repos/private/panel/.venv/lib/python3.11/site-packages/param/parameterized.py", line 530, in _f
    return f(self, obj, val)
           ^^^^^^^^^^^^^^^^^
  File "/home/jovyan/repos/private/panel/.venv/lib/python3.11/site-packages/param/parameterized.py", line 1498, in __set__
    self._validate(val)
  File "/home/jovyan/repos/private/panel/.venv/lib/python3.11/site-packages/param/parameters.py", line 1192, in _validate
    self._validate_length(val, self.length)
  File "/home/jovyan/repos/private/panel/.venv/lib/python3.11/site-packages/param/parameters.py", line 1185, in _validate_length
    raise ValueError(
ValueError: Attribute 'length' of Tuple parameter 'Map.bounds' is not of the correct length (0 instead of 2).

[MarcSkovMadsen]

FIXED

Pydantic seems to missing as a dependency for the docs

[MarcSkovMadsen]

FIXED

Ahh. The dataclass functionality for Pydantic does not add specific parameter types yet.

[MarcSkovMadsen]

Fixed by adding ModelForm

Our code instantiates pydantic models. Often they don't have default values. Instead initial values are required.

This makes it a bit hard to use our features. Especially for creating forms with validation which a popular use case (c.f. pydantic-panel, streamlit-pydantic, dash-pydantic-form).

For example the below test will currently raise an exception

def test_to_parameterized_no_defaults():
    from pydantic import BaseModel
    class ExampleModel(BaseModel):
        some_text: str
        some_number: int
    
    class ExampleModelParameterized(ModelParameterized):
        _model_class = ExampleModel

    ExampleModelParameterized()

Something like the default_values of the below code is currently required to work around this

import panel as pn
from pydantic import BaseModel
import param
from panel._dataclasses.pydantic import PydanticUtils

pn.extension()

class ModelForm(pn.viewable.Viewer):
    value = param.ClassSelector(class_=BaseModel, allow_None=True)

    submit_button_visible = param.Boolean(default=True, label="Show Submit Button")

    def __init__(self, model_class, submit_button_visible: bool=True, **params):
        self._model_class = model_class
        self._fields = list(model_class.model_fields.keys())
                
        super().__init__(**params)

        fields = model_class.model_fields
        default_values = {field: PydanticUtils.create_parameter(model_class, field).default for field, info in fields.items() if info.is_required()}
        model=model_class(**default_values)
        self._model = model=model_class(**default_values)
        parameters = list(ExampleModel.model_fields.keys())
        parameterized = pn.dataclass.to_viewer(model)
        parameterized.param.watch(self._update_value_on_parameter_change, parameters)
        submit = pn.widgets.Button(name="Submit", button_type="primary", on_click=self._update_value, visible=self.param.submit_button_visible)
        
        self._form = pn.Column(
            pn.Param(parameterized, parameters=parameters),
            submit)

    def _update_value(self, *args):
        self.value = self._model.copy(deep=True)

    def _update_value_on_parameter_change(self, *args):
        if not self.submit_button_visible:
            self.value = self._model.copy(deep=True)

    def __panel__(self):
        return self._form

    @param.depends("value")
    def value_as_dict(self):
        if not self.value:
            return {}
        return self.value.dict()

class ExampleModel(BaseModel):
    some_text: str
    some_number: int
    some_boolean: bool

form = ModelForm(model_class=ExampleModel)
    

pn.Column(form, pn.pane.JSON(form.value_as_dict), form.param.submit_button_visible).servable()

[MarcSkovMadsen]

I've tried to achieve almost feature parity with pydantic-panel.

What is missing is Pydantic BaseModel attributes and pandas intervals.

• BaseModels will be added later
• I did not understand the pandas intervals support.

pip install pydantic-panel

import pydantic
import panel as pn
from typing import List
from pydantic_panel.dispatchers import infer_widget
from datetime import datetime, date
import numpy as np
import pandas as pd

pn.extension("tabulator")

class ChildModel(pydantic.BaseModel):
    name: str = "child"

class SomeModel(pydantic.BaseModel):
    name: str = "some model"
    
    child_field: ChildModel = ChildModel()
    date_field: date = date(2024,1,2)
    dateframe: pd.DataFrame = pd.DataFrame({"x": [1], "y": ["a"]})
    datetime_field: datetime = datetime(2024,1,1)
    dict_field: dict = {"a": 1}
    float_field: float = 42
    int_field: int = pydantic.Field(default=2, lt=10, gt=0, multiple_of=2)
    list_field: list = [1, "two"]
    nparray_field: np.ndarray = np.array([1, 2, 3])
    str_field: str = pydantic.Field(default = "to", min_length=2, max_length=10)
    tuple_field: tuple = ("a", 1)

    class Config:
        arbitrary_types_allowed = True # to allow np.array

model = SomeModel()

pydantic_panel_editor = pn.panel(model, sizing_mode="fixed") # Pydantic(model).layout[0]
print(type(pydantic_panel_editor))
panel_editor = pn.Param(pn.dataclass.to_parameterized(model))

pn.Row(
    pydantic_panel_editor,
    panel_editor,
).servable()

[MarcSkovMadsen]

What else should be done here?

[philippjfr]

I'm still fully on board with the aims of this PR but it's simply too large a PR to make it into 1.5.0 at this point.

[grepler]

Peanut gallery over here, I'm keen to see interoperability between Pydantic, but note that this hasn't made it's way onto newer roadmaps - is this still a possibility?

[jbednar]

We're all still very interested, but haven't had time to work on it!

[philippjfr]

Peanut gallery over here, I'm keen to see interoperability between Pydantic

It is very much on the roadmap, but it will likely start with a pydantic interop in Param, which we can then build on in Panel.