`xsdata == 24.4` causing issues with `to_xml` downstream

[erickmartins]

As of 11h ago xsdata has a new version (24.4) and to_xml breaks spectacularly in omero-cli-transfer now. I can confirm that the same exact build ran 2 days ago with no problem (see both attempts here if it helps!). As far as I can see xsdata version is the only change. Full error stack below if it helps, from the to_xml call onwards.

error stack

src/generate_xml.py:1000: in populate_xml
    print(to_xml(ome), file=fp)
/tmp/miniconda/envs/omero/lib/python3.9/site-packages/ome_types/_conversion.py:316: in to_xml
    xml = serializer.render(obj, ns_map=ns_map)
/tmp/miniconda/envs/omero/lib/python3.9/site-packages/xsdata/formats/dataclass/serializers/xml.py:42: in render
    self.write(output, obj, ns_map)
/tmp/miniconda/envs/omero/lib/python3.9/site-packages/xsdata/formats/dataclass/serializers/xml.py:59: in write
    handler.write(events)
/tmp/miniconda/envs/omero/lib/python3.9/site-packages/xsdata/formats/dataclass/serializers/mixins.py:138: in write
    for name, *args in events:
/tmp/miniconda/envs/omero/lib/python3.9/site-packages/xsdata/formats/dataclass/serializers/mixins.py:374: in generate
    yield from self.convert_dataclass(obj, qname=qname, xsi_type=xsi_type)
/tmp/miniconda/envs/omero/lib/python3.9/site-packages/xsdata/formats/dataclass/serializers/mixins.py:419: in convert_dataclass
    yield from self.convert_value(value, var, namespace)
/tmp/miniconda/envs/omero/lib/python3.9/site-packages/xsdata/formats/dataclass/serializers/mixins.py:489: in convert_value
    yield from self.convert_list(value, var, namespace)
/tmp/miniconda/envs/omero/lib/python3.9/site-packages/xsdata/formats/dataclass/serializers/mixins.py:510: in convert_list
    yield from self.convert_value(value, var, namespace)
/tmp/miniconda/envs/omero/lib/python3.9/site-packages/xsdata/formats/dataclass/serializers/mixins.py:491: in convert_value
    yield from self.convert_any_type(value, var, namespace)
/tmp/miniconda/envs/omero/lib/python3.9/site-packages/xsdata/formats/dataclass/serializers/mixins.py:569: in convert_any_type
    yield from self.convert_xsi_type(value, var, namespace)
/tmp/miniconda/envs/omero/lib/python3.9/site-packages/xsdata/formats/dataclass/serializers/mixins.py:452: in convert_xsi_type
    yield from self.convert_dataclass(
/tmp/miniconda/envs/omero/lib/python3.9/site-packages/xsdata/formats/dataclass/serializers/mixins.py:410: in convert_dataclass
    for key, value in self.next_attribute(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cls = <class 'xsdata_pydantic_basemodel.bindings.XmlSerializer'>
obj = Plate(
   id='Plate:3',
   name='Plate Name 0',
   description='Plate 0 of 2',
   wells=[<1 field_type>],
   annotation_refs=[{'id': 'Annotation:17148068931407003905'}, {'id': 'Annotation:17148068931407003910'}],
)
meta = XmlMeta(clazz=<class 'ome_types._autogenerated.ome_2016_06.plate.Plate'>, qname='{http://www.openmicroscopy.org/Schema...)}, any_attributes=[], wrappers={}, namespace='http://www.openmicroscopy.org/Schemas/OME/2016-06', mixed_content=False)
nillable = False, xsi_type = None, ignore_optionals = False
ignore_unset = False, attribute_sort_key = None

    @classmethod
    def next_attribute(
        cls,
        obj: BaseModel,
        meta: XmlMeta,
        nillable: bool,
        xsi_type: str | None,
        ignore_optionals: bool,
        ignore_unset: bool = False,
        attribute_sort_key: Callable | None = None,
    ) -> Iterator[tuple[str, Any]]:
        """
        Return the attribute variables with their object values if set and not
        empty iterables.

        :param obj: Input object
        :param meta: Object metadata
        :param nillable: Is model nillable
        :param xsi_type: The true xsi:type of the object
        :param ignore_optionals: Skip optional attributes with default
            value
        :return:
        """

        set_fields = obj.model_fields_set if ignore_unset else set()
        vars_ = meta.get_attribute_vars()
        if attribute_sort_key is not None:
            vars_ = sorted(meta.get_attribute_vars(), key=attribute_sort_key)

        # ^^^ new

        for var in vars_:
            if var.is_attribute:
                value = getattr(obj, var.name)
                if (
                    value is None
                    or (collections.is_array(value) and not value)
                    or (ignore_optionals and var.is_optional(value))
                    or (ignore_unset and var.name not in set_fields)  # new
                ):
                    continue
>               yield var.qname, cls.encode(value, var)
E               AttributeError: type object 'XmlSerializer' has no attribute 'encode'

/tmp/miniconda/envs/omero/lib/python3.9/site-packages/xsdata_pydantic_basemodel/bindings.py:144: AttributeError

[tlambert03]

Thanks @erickmartins, yeah I just started seeing the errors pop up in my ci cron jobs this morning. I'll get on it soon, but am away teaching a course so things may be a bit slow. Xsdata has been changing a lot recently, I'm trying not to part hard upper pins, but may just do that until things settle down

[erickmartins]

absolutely! just wanted to make sure to flag it and provide as much info as I could.

[tlambert03]

i just pushed a v0.5.1.post1 to pypi that pins to <24.4 for now

[joshmoore]

Thanks, @tlambert03. I can confirm that mamba install -c conda-forge 'xsdata<24.4'ing, solved the problem below. (24.3.1 is what got installed)

xsdata error

$ xsdata generate ../ome-types/src/ome_types/ome-2016-06.xsd --output pydantic-basemodel

  File "/Users/jamoore/opt/ome_types_ld/second-attempt/generated/__init__.py", line 1, in <module>
    from generated.ome_2016_06 import (
  File "/Users/jamoore/opt/ome_types_ld/second-attempt/generated/ome_2016_06.py", line 573, in <module>
    class TiffData(BaseModel):
  File "/Users/jamoore/opt/ome_types_ld/second-attempt/generated/ome_2016_06.py", line 653, in TiffData
    class Uuid(BaseModel):
  File "/Users/jamoore/opt/ome_types_ld/second-attempt/generated/ome_2016_06.py", line 666, in Uuid
    value: str = Field(
                 ^^^^^^
  File "/Users/jamoore/micromamba/envs/ome-types2/lib/python3.12/site-packages/pydantic/fields.py", line 751, in Field
    raise PydanticUserError('`regex` is removed. use `pattern` instead', code='removed-kwargs')
pydantic.errors.PydanticUserError: `regex` is removed. use `pattern` instead

Edit: it persists though on import generated. ⚠️ All versions back to 23.5 show the same behavior. (22.x reduces the ome-types version leading to a different issue.)

[tlambert03]

@joshmoore, just to check, were you installing ome-types from conda? I did fix it on pypi (https://pypi.org/project/ome-types/) ... but need to update the feedstock still

[tlambert03]

will fix and merge https://github.com/conda-forge/ome-types-feedstock/pull/20 today

[joshmoore]

@tlambert03: the installs were with conda (micromamba) but even manual downgrades throw. Just not during $ xsdata but during import generated.

[tlambert03]

does the new install from conda work fine?

[joshmoore]

Unfortunately no.

[tlambert03]

i'm sorry @joshmoore, I wasn't paying close enough attention to your error here. I assumed it was the same as @erickmartins just for conda. I see now that you're generating your own bindings to the xsd, and not using ome-types... so this is more about the --output pydantic-basemodel flag in xsdata generate and compatibility with various versions of pydantic.

Note that in ome-types, I use pydantic-compat for broader compatibility with all versions of pydantic. So, if you are generating your own bindings using xsdata generate you'll likely need to do a lot more work (see, for example all the stuff in the ome_autogen folder here). If you just want to move past this error, you can use

from pydantic_compat import BaseModel, Field

at the top of your ome_2016_06.py file

[joshmoore]

Thanks, @tlambert03! And no worries. I'm slightly misusing the ticket anyway :wink: