(🐞) false positive serialize error with union of `list`s

[KotlinIsland]

Initial Checks

• [X] I confirm that I'm using Pydantic V2

Description

pydantic_core._pydantic_core.PydanticSerializationError: Pydantic serializer warnings:
  PydanticSerializationUnexpectedValue: Expected `bool` but got `int` with value `2` - serialized value may not be as expected
  PydanticSerializationUnexpectedValue: Expected `str` but got `int` with value `2` - serialized value may not be as expected

Example Code

from pydantic import TypeAdapter, BaseModel

TypeAdapter(list[bool | str] | list[int]).dump_python([2], warnings="error")

Python, Pydantic & OS Version

pydantic version: 2.9.2
        pydantic-core version: 2.23.4
          pydantic-core build: profile=release pgo=false
                 install path: C:\Users\AMONGUS\projects\test\.venv\Lib\site-packages\pydantic
               python version: 3.12.2 (tags/v3.12.2:6abddd9, Feb  6 2024, 21:26:36) [MSC v.1937 64 bit (AMD64)]
                     platform: Windows-10-10.0.19045-SP0
             related packages: mypy-1.11.2 typing_extensions-4.12.2
                       commit: unknown

[sydney-runkle]

Looks like a bug in pydantic-core, I can take a look later this week.

[sanderr]

I encountered this issue as well. I expect you are already aware of the cause, but for the sake of completeness I'll say that this seems to be a regression in pydantic-core 2.23.4, with pydantic/pydantic-core#1449 as the likely cause.

[sydney-runkle]

@sanderr, any interest in opening a PR with a fix?

[sanderr]

I can't make any promises at the moment but I can probably make some time for it sometime this week. I'll report here if/when I get started on it.

I do have to confess I have little experience with Rust outside of some learning exercises, but the affected code surface looks simple enough.

How do you see the intended behavior? Report line-by-line as in pydantic/pydantic-core#1449, but only if all unions come up empty?

[sydney-runkle]

How do you see the intended behavior? Report line-by-line as in https://github.com/pydantic/pydantic-core/pull/1449, but only if all unions come up empty?

Right - only if validation against all union members fails

[sanderr]

Before I get started on an implementation attempt, can you confirm that conceptually it would look as follows?

Union currently logs errors for its elements here (iff none of them match). Since the union might be nested in another union, this is not always appropriate. Instead, iff extra.check.enabled() (indicating we're below a Union), we should propagate the errors upwards somehow, and not log them. Then only the outer one would make the actual decision to log or not to log element errors.

I haven't really figured out the upwards propagation yet. I'm thinking of some error grouping, but I have yet to check if that fits. Or perhaps add a field to Extra for tentative union warnings?

[sydney-runkle]

Hi @sanderr,

Thanks for following up. Let me circle back with the team tomorrow morning and I'll get back to you!

[sanderr]

@sydney-runkle did anything come out of the discussion with the team regarding this?

[sydney-runkle]

Hi! Yes! We've fixed things here :)

No errors are raised when testing against v2.10.0b2.

Going to close as resolved for now, feel free to ping me if you're still encountering any issues.

[sydney-runkle]

See https://github.com/pydantic/pydantic-core/pull/1513 for the fix :)

[sanderr]

Thanks for the follow-up!