pydantic Add support for `typing.Never`

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description

I get an error. I understand that it should work without Never, but I still want to point it out sensitively.

pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for typing.Never. Set `arbitrary_types_allowed=True` in the model_config to ignore this error or implement `__get_pydantic_core_schema__` on your type to fully support it.
If you got this error by calling handler(<some type>) within `__get_pydantic_core_schema__` then you likely need to call `handler.generate_schema(<some type>)` since we do not call `__get_pydantic_core_schema__` on `<some type>` otherwise to avoid infinite recursion.
For further information visit https://errors.pydantic.dev/2.7/u/schema-for-unknown-type

Example Code

class PersonFilmRoles(BaseModel):
    id: str
    roles: list[str | Never]

Python, Pydantic & OS Version

pydantic version: 2.7.4
pydantic-core version: 2.18.4
python version: 3.12.3

Jun 22 '24 15:06 ArtemIsmagilov

Hi @ArtemIsmagilov,

Thanks for reporting this. I'd consider this more of a feature request - I can't really understand a practical example of where you'd want to use the typing.Never hint for a model field - could you shed some light on that?

If there's not a practical example, I think we'll probably mark this as not planned.

Jun 24 '24 13:06 sydney-runkle

Hi, @sydney-runkle, I want to specify a model that would accept only an empty list, without elements. to do this, I'm creating the following class. For example, I can use this validator for the API

Newbie actor at first job, he shouldn't have roles

class PersonFilmRoles(BaseModel):
    id: str
    roles: list[Never]

newbie_actor = PersonFilmRoles(id='id', roles=[])

Jun 24 '24 13:06 ArtemIsmagilov

Hmm,

For this case, I'm still not convinced, because I feel as though

from typing import Annotated

from pydantic import BaseModel
from annotated_types import MaxLen


class PersonFilmRoles(BaseModel):
    id: str
    roles: Annotated[list, MaxLen(0)]


newbie_actor = PersonFilmRoles(id='id', roles=[])

Would be more appropriate.

That being said, thinking about it more, perhaps it makes sense as an attribute annotation if you seek to distinguish a more abstract base class from a concrete one, so I'll leave this as a feature request.

Feel free to open a PR adding support!

Jun 24 '24 13:06 sydney-runkle

To be honest, I don't understand you a little. The standard Never type in pydantic does not work for you. Is this really not a bug but a feature?

Jun 24 '24 13:06 ArtemIsmagilov

Feel free to open a PR adding support!

Thanks)

Jun 24 '24 13:06 ArtemIsmagilov

I suppose the categorization here isn't super critical - it's low priority either way, but we welcome PRs from the community :)

Jun 24 '24 14:06 sydney-runkle

I'm not sure I really understand the use case here. Never is the bottom type, it has a specific meaning in the typing spec. Using it to indicate that a list has no element does not really follow its semantics. At the very least, I think it could make sense to have:

from typing import Never

class Model(BaseModel):
    foo: Never

Model(foo=...)  # Always fails, no matter the given value of `foo`

Jun 27 '24 10:06 Viicos

Xref https://github.com/pydantic/pydantic-core/issues/619

Aug 19 '24 08:08 davidhewitt

To offer a practical use case, I've found Never handy in "knocking out" a field from a compound type for the purposes of writing @overload signatures. When a field's type comes from a type parameter, instantiating the containing structure with that parameter set to Never creates a type in which that field must be absent (or, as in the example below, must be None).

H = TypeVar('H', bound=DriveCore)

class HyperdriveConfig(BaseModel, Generic[H]):
    core: H | None = None
    
    antimatter_containment_pressure: float = 8.1e41
    flux_capacitance_ratio: float = 13.41
    graviton_polarization_angle: float = 0.0

    # ... 106 other fields ...

    def recalibrate(self, updates: HyperdriveConfig[Never]) -> Self:
        return self.model_copy(update=updates.model_dump())


drive = HyperdriveConfig(core=StandardCore(), flux_capacitance_ratio=-1.8)

# these are fine because they don't replace the core with an incompatible type
drive.recalibrate(HyperdriveConfig(antimatter_containment_pressure=1e42)) 
drive.recalibrate(HyperdriveConfig(core=None)) 

# ...but thanks to Never, this will not type check.
drive.recalibrate(HyperdriveConfig(core=DangerouslyIncompatibleCore()))

The above will (presumably) cause a runtime error from Pydantic, but it would be nice if this worked. I landed here after trying to do something similar in a real project, and would make use of it if it were supported.

Sep 02 '24 21:09 scullion

Hum, I'm not sure Never fulfills your use case. If you only want HyperdriveConfig instances without any core set, the signature should look like recalibrate(self, updates: HyperdriveConfig[None]) -> Self.

Sep 03 '24 07:09 Viicos

If you only want HyperdriveConfig instances without any core set, the signature should look like recalibrate(self, updates: HyperdriveConfig[None]) -> Self.

That wouldn't type check because None is not compatible with the type bound on H. The bound would have to be relaxed to include None.

H = TypeVar('H', bound=DriveCore | None)

But that necessitates further complications if H is used in other places where None must be prohibited: now a second TypeVar must be introduced.

And even that doesn't help if None is not a permissible value for the field.

core: H = Field(default_factory=IdleCore)

In that case a new type, compatible with the bound, has to be invented as a substitute for Never.

class NeverCore(DriveCore): ...
ConfigWithoutCore = HyperdriveConfig[NeverCore]

This is the workaround used in the code that inspired this example.

Sep 03 '24 10:09 scullion

I'd like to try this for v2.10 :).

Sep 04 '24 09:09 sydney-runkle

@scullion I see. But your use case might be a bit different: Never is the bottom type, meaning Never | <tp> ~ <tp>. In your example:

from a type checking perspective (i.e. during static analysis), HyperdriveConfig[Never] means any instance of HyperdriveConfig where core is None (and only None) can be used, because Never | None ~ None.
from a Pydantic perspective (i.e. runtime analysis), HyperdriveConfig[Never] will create a new Pydantic with a new core schema generated. During the creation of this new model, type variables will be "replaced" by the provided type, but no simplification of the type is performed. Pydantic will thus try to generate a schema for the field core of type Never | None, and will fail because it doesn't know about Never.

Pydantic does not currently do any simplification of types because generally the user can do it when defining a model (i.e. there's no reason to define a field with type Any | None, just use Any). However, it could make sense when parametrizing generics. I think it is reasonable to add support for Any | <tp> (~ Any) and Never | <tp> (~ <tp>). This would solve your issue without having to add support for Never yet.

Sep 04 '24 12:09 Viicos

There seems to be some resistance to supporting Never 🙂 If that comes from a desire not to add code that "should never need to be run", I appreciate that, and can only say that the always-fails validator proposed in https://github.com/pydantic/pydantic-core/issues/619 would serve the same pupose as error handling code a Pydantic user is currently obliged to write if they define a dummy "never type" to work around the lack of Never support, such as the NeverCore constructor in the example below.

What @Viicos propses solves my real life problem, but doesn't handle the case where the type var isn't used as part of a union as shown below. Currently if the user has a field typed T rather than T | None, they are obliged to invent a type compatible with T like NeverCore to achive an effect similar to Never.

H = TypeVar("H", bound=DriveCore)
NewH = TypeVar("NewH", bound=DriveCore)

class NeverCore(DriveCore):
    def __init__(self):
        raise RuntimeError("something is quite seriously wrong")

class HyperdriveConfig(BaseModel, Generic[H]):
    core: H = Field(default_factory=IdleCore)

    def recalibrate(self, updates: HyperdriveConfig[NeverCore]) -> Self:
        return self.model_copy(update=updates.model_dump())
    
    def install_core(self, core: NewH) -> HyperdriveConfig[NewH]:
        self.core.safe_power_down()
        return self.model_copy(update={'core': core})
    
# it would be dangerous to replace the core during a recalibration
drive.recalibrate(HyperdriveConfig(core=FluxGravitonCore())) # type error

# ... instead should to use install_core()
drive.install_core(FluxGravitonCore()) # ok

Isn't having Pydantic do type algebra on unions harder to implement than just having an always-fails validator? Either would solve my use case, but as a user I'd prefer first class support for Never purely on the basis of the principle of least surprise.

Sep 04 '24 14:09 scullion

There seems to be some resistance to supporting Never

Not at all! I'm just stating that your use case can be treated differently. To be clear, adding support for Never will also solve your use case (and as Sydney said we will try to tackle it for 2.10). Never | None will be supported when building the schema, and validation will work as expected. That is, for a given value v:

v will try to be validated against Never, and fails because Never should always fail validation.
v will try to be validated against None, and will fails if v is not None.

But having "type algebra on unions" just for this case just makes things cleaner imo. The resulting core schema is smaller and probably easier to understand. The generated JSON Schema might be simplified as well.

Sep 04 '24 14:09 Viicos

This would be a cool feature for sure. I've bumped this to v2.11 - PRs welcome with some help here! Not quite high priority enough to fit into our current sprint.

Oct 07 '24 15:10 sydney-runkle

This is lower priority than I thought - bumped out of milestone. PRs still welcome, of course.

Oct 28 '24 17:10 sydney-runkle

Thanks for keeping me updated on this feature. I'll see if I have the skills to do it.

Oct 28 '24 17:10 ArtemIsmagilov

@ArtemIsmagilov, note that if this is going to be implemented, it will be following the semantics specified in https://github.com/pydantic/pydantic/issues/9731#issuecomment-2194319078 (just wanted to make sure you were aware of it).

Oct 28 '24 19:10 Viicos

Hi y'all! I have, potentially, another need for Never support. In SQLModel, im representing a database type that has no meaning in python, and must never be accessed - neither selected, nor inserted. I would love to represent it as:

from sqlmodel import SQLModel, Field
from sqlalchemy.types import UserDefinedType

class Earth(UserDefinedType):
    def get_col_spec(self, **kwargs):
        return "earth"

class MyTable(SQLModel, table=True):
    earth: Never = Field(sa_type=Earth)

The idea is that while it makes sense to index it or run sql-specific queries on it, it has no meaning ouside of its own earthdistance extension

Nov 14 '24 16:11 iliya-malecki