Silent data loss when parsing an iterator into a union of sequences

[jcrist]

Initial Checks

• [X] I confirm that I'm using Pydantic V2

Description

Given a field that's a union of sequences (where pydantic tries validating against each sequence type in turn), if an iterator is passed in that fails to match the first type in the union, pydantic will silently drop all data previously seen in the iterator as it continues to try to match against the next type.

I've checked and this bug is new in Pydantic V2, the following code passes fine on Pydantic V1

Example Code

from pydantic import BaseModel
from typing import Union


class Test(BaseModel):
    # ints first
    x: Union[list[int], list[str], None] = None
    # strs first
    y: Union[list[str], list[int], None] = None


def gen():
    for val in range(3):
        yield val


# Passes, since generator is only iterated over once
assert Test(x=gen()) == Test(x=[0, 1, 2])

# Fails, since the generator is iterated over twice
assert Test(y=gen()) == Test(y=[0, 1, 2])

Python, Pydantic & OS Version

pydantic version: 2.6.0
        pydantic-core version: 2.16.1
          pydantic-core build: profile=release pgo=true
                 install path: /home/jcristharif/miniconda3/envs/msgspec/lib/python3.9/site-packages/pydantic
               python version: 3.9.18 | packaged by conda-forge | (main, Dec 23 2023, 16:33:10)  [GCC 12.3.0]
                     platform: Linux-5.15.0-92-generic-x86_64-with-glibc2.35
             related packages: pyright-1.1.344 typing_extensions-4.9.0 mypy-1.8.0
                       commit: unknown

[sydney-runkle]

@jcrist,

Thanks for reporting this. This is indeed an interesting bug! I'm guessing that fixing this will require some changes to pydantic-core.

Marking this as a bug, PRs welcome with a fix if anyone is interested in contributing!