DefaultDict default function gets replaced with type creator in object creation

[edelooff]

Checks

• [x] I added a descriptive title to this issue
• [x] I have searched (google, github) for similar issues and couldn't find anything
• [x] I have read and followed the docs and still think this is a bug

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.8.2
            pydantic compiled: True
                 install path: /usr/local/lib/python3.9/site-packages/pydantic
               python version: 3.9.4 (default, Apr  5 2021, 01:50:46)  [Clang 12.0.0 (clang-1200.0.32.29)]
                     platform: macOS-11.6-x86_64-i386-64bit
     optional deps. installed: ['typing-extensions']

from collections import defaultdict
from typing import DefaultDict

import pydantic

class Model(pydantic.BaseModel):
    values: DefaultDict[str, int]

example = Model(values=defaultdict(lambda: 6, {"low": 3, "high": 9}))
assert example.values["low"] == 3  # passes
assert example.values["high"] == 9  # passes
assert example.values["medium"] == 6, "defaultdict default function not retained"  # fails

This breaks the constant_factory example as provided on https://docs.python.org/3/library/collections.html#defaultdict-examples as otherwise legitimate and useful use of defaultdict. I was hoping to use this in a situation where there are different default values for different instances of the model, so source-level constants do not work.

[edelooff]

This could be worked around by means of a validator, adding an explicit "default" key to the dict to pop off and construct a new defaultdict from:

class Model(pydantic.BaseModel):
    values: DefaultDict[str, int]

    @pydantic.validator("values")
    def _construct_defaultdict(cls, value):
        if "default" not in value:
            raise ValueError("No 'default' provided")
        default = value.pop("default")
        return defaultdict(lambda: default, value)


example = Model(values={"low": 3, "high": 9, "default": 6})
assert example.values["medium"] == 6  # Success

However, this fails if the dictionary key is constrainted by pydantic.constr. Regex validation for example would have to explicitly include a |default portion, or similar.

[MaximilianFranz]

Adding to this, the following defaultdict typing also is not possible:

class ModelWithDD(BaseModel):
    some_dict: DefaultDict[str, List[int]]


dict_as_default = defaultdict(list)
dict_as_default.update({"some_key": [1, 2, 3]})
instance = ModelWithDD(some_dict=dict_as_default)

assert isinstance(instance.some_dict, defaultdict)
assert len(instance.some_dict["does_not_exist"]) == 0
>> TypeError: Type List cannot be instantiated; use list() instead

I am solving this internally with:

    class ModelWithDD(ImmutableBaseModel):
        some_dict: DefaultDict[str, List[int]]

        @validator("some_dict")
        def make_default_dict(cls, value):
            new_dict = defaultdict(list, value)
            return new_dict

which is acceptable, since we can't assume pydantic to know what defaultdict default callable we would like to use. Theoretically, the callable could be derived from the type-hint.

[dmontagu]

In principle we should be able to provide a better experience for DefaultDict in v2, someone just needs to implement it.

I would suggest that we implement an annotation that lets you provide the default factory to use when validating a defaultdict from something that would otherwise successfully validate as a dictionary, and if that annotation is missing, raise an error indicating that it should be used.

It may also be a good idea to, at the same time, support subclasses of DefaultDict and properly reflect their generic parameters.

Open to PRs for this, otherwise I'm sure we'll get to it eventually.