pydantic compatible types

[xneanq]

Hi I was fiddling around trying to implement pydantic compatible subclasses of whenever classes potentially to add to pydantic-extra-types but they're marked @final

What's the reason for them to be @final?

Any chance you would implement pydantic methods or lift the @final restriction?

[ariebovenberg]

Hi @xneanq, thanks for opening this issue.

Regarding @final: they are marked this way for two reasons:

1. In my experience, designing a class for subclassing requires special attention (i.e. documenting what you can override, what parts are stable, method call order, etc.). Since all classes in whenever are designed as concrete classes, it was most straightforward to simply mark them as "final". It prevents subclassing since the classes aren't designed for it.
2. The rust extension is able to optimize performance in some cases because there are no unexpected subclasses.

That said, Pydantic compatibility seems like a good idea, considering its popularity 👍 . What would needed to be implement in order to be pydantic-compatible? Perhaps some kind of patching can be done 🤔

[illphil]

looking at the pendulum_dt.py in pydantic-extra-types it looks like a few methods would need to be implemented which look fairly simple

I'm not sure the neatest was to proceed, my first thoughts revolve around a mixin class implementing the required methods conditional on pydantic being available perhaps via extras

[ariebovenberg]

At the moment, my preferred way would be to add the required methods directly to the whenever library.

This may take some time to implement though, as I want to do make sure it integrates well with pydantics customization options.

If you need pydantic support today, the best solution would be to use Annotated, as described here

[ariebovenberg]

Looking at pendulum_dt more closely, you should be able to build a similarly functioning whenever plugin without subclassing:

# pydantic_extra_types/whenever.py
from whenever import LocalDateTime as _LocalDateTime
from typing import Annotated

class _LocalDateTimePydanticAnnotation:
    @classmethod
    def __get_pydantic_core_schema__(...): ...
    @classmethod
    def __get_pydantic_json_schema__(...): ...

LocalDateTime = Annotated[_LocalDateTime, _LocalDateTimePydanticAnnotation]

the result:

from pydantic_extra_types.whenever import LocalDateTime

class MyModel(Model):
    x: LocalDateTime

# still usable as a class
LocalDateTime.MIN  # still works
LocalDateTime(2020, 1, 1)  # also works

note that in the long term, it'd always be nicer to do:

from whenever import LocalDateTime

class MyPydanticModel(Model):
    x: LocalDateTime  # usable directly

[illphil]

sounds like you've got the next release in the bag ;)

[ariebovenberg]

@illphil well, I suppose it's a two-phase thing:

1. add the Annotated thing to pydantic_extra_types. @illphil @xneanq feel free to take this up. No changes to whenever needed.
2. in the long term, add support to whenever directly. However, seeing as option (1) is so reasonable, it's unsure if this would even be needed.

[dusktreader]

@ariebovenberg I've opened this PR in pydantic-extra-types to add in annotated types for most of whenever's tyeps: https://github.com/pydantic/pydantic-extra-types/pull/310

[ariebovenberg]

Hi @dusktreader thanks but maybe wait off on this until the next release of whenever, which includes some relevant changes and may even add pydantic support built-in

[dusktreader]

@ariebovenberg: OK. I will close the PR.

[ariebovenberg]

@dusktreader sorry for the confusion 😬 . Looking at your PR, it looks like we can reuse most of it 👍 . I'm mostly AFK this week, so I can look it over in detail later.

[ariebovenberg]

@dusktreader looking at your PR, pydantic support is quite a bit more complex than I thought. Perhaps you can help me understand some pydantic_core's validation API:

• Is it customary for a validator to "try everything until something works"? i.e. in your example, we first try ISO, then RFC2822. What's to stop from trying dateutil.parse afterwards? I'm not saying we should, but is where to draw the line? Pydantic has quite idiosyncratic some logic to how it parses datetimes. My own preference is to keep things simple—just parsing ISO. Or do many people really need to "parse whatever"?
• In your PR, you use no_info_wrap_validator_function but don't seem to use the "inner" handler. Could we just use no_info_plain_validator_function to simplify? Or is there some other reason to use the wrapping validator, perhaps it sets some useful metadata?
• Is it necessary to support pydantic v1? Or is everybody already moved to V2?

[ariebovenberg]

Also note: pydantic support won't make it into the very next release (0.8) because it's already quite big as-is. Pydantic support is near the top of the list for features after this though

[dusktreader]

@dusktreader looking at your PR, pydantic support is quite a bit more complex than I thought. Perhaps you can help me understand some pydantic_core's validation API:

Sorry for the long delay in getting back to you. I was finishing up the final rounds of interviewing for a few different companies. The good news is that I got an offer and have a week before the new job starts to catch up!

• Is it customary for a validator to "try everything until something works"? i.e. in your example, we first try ISO, then RFC2822. What's to stop from trying dateutil.parse afterwards? I'm not saying we should, but is where to draw the line? Pydantic has quite idiosyncratic some logic to how it parses datetimes. My own preference is to keep things simple—just parsing ISO. Or do many people really need to "parse whatever"?

The idea here is to offer a little more flexibility. Since whenever knows a few different formats, it seems like it might improve usability to exhaust the known formats before giving up. Before coming to whenever, I was a big fan of Pendulum which has a parse() function that will try a few different formats before giving up. I found it very useful. I think that kind of flexibility contributed to the success of Pendulum and Arrow, because folks could usually get what they needed by calling a single function.

• In your PR, you use no_info_wrap_validator_function but don't seem to use the "inner" handler. Could we just use no_info_plain_validator_function to simplify? Or is there some other reason to use the wrapping validator, perhaps it sets some useful metadata?

I did some digging into this. It does work with no_info_plain_validator_function(), and all the tests pass. However, for some reason, that function's json_schema_input_schema keyword argument isn't working. So any JSON schema generated from a model with a whenever field wouldn't have any indicators that the field is a datetime. That may not matter since the whenver fields aren't constrained to JSON's preferred 8601 format. That's something to consider, though. I think I will file a bug report about the kwarg not working as expected.

• Is it necessary to support pydantic v1? Or is everybody already moved to V2?

I think the overhead of supporting pydantic v1 would be prohibitive. I think people are being pushed toward v2 with some gusto. I think it's safe to only target v2, and that would keep the scope more focused.

[ariebovenberg]

The idea here is to offer a little more flexibility. Since whenever knows a few different formats, it seems like it might improve usability to exhaust the known formats before giving up. Before coming to whenever, I was a big fan of Pendulum which has a parse() function that will try a few different formats before giving up. I found it very useful. I think that kind of flexibility contributed to the success of Pendulum and Arrow, because folks could usually get what they needed by calling a single function.

Agree that this could be useful—so long as there are "strict" interfaces too. One thing I've considered is adding a parse_dateutil method that delegates to dateutil.parser to parse pretty much anything. This is what pendulum.parse(x, strict=False) does. Related: #174

I did some digging into this. It does work with no_info_plain_validator_function(), and all the tests pass. However, for some reason, that function's json_schema_input_schema keyword argument isn't working. So any JSON schema generated from a model with a whenever field wouldn't have any indicators that the field is a datetime. That may not matter since the whenver fields aren't constrained to JSON's preferred 8601 format. That's something to consider, though. I think I will file a bug report about the kwarg not working as expected.

Thanks for looking into this. What I'll probably do for now is add "just enough" pydantic support for serilializing to ISO and back—this covers a big chunk of pydantic use cases. In the future I'll consider whether to broaden the accepted formats.

PS: Congrats on the new job; looks like you still need to update your GH profile though 👀

[ariebovenberg]

@dusktreader OK so I've implemented what I think is enough for basic pydantic serialization/deserialization support, and I've released it as 0.8.1rc1. Can you have a look if it works?

Essentially the implementation was simply this for all classes:

    core_schema.no_info_plain_validator_function(
        lambda v: v if type(v) is cls else cls.parse_common_iso(v),
        serialization=core_schema.plain_serializer_function_ser_schema(
            cls.format_common_iso,
            when_used="json-unless-none",
        ),
    )

[AllanDaemon]

OK so I've implemented what I think is enough for basic Pydantic serialization/de-serialization support, and I've released it as 0.8.1rc1. Can you have a look if it works?

I just tested it bumping the version from 0.8.0 to 0.8.1rc1. Now my program crashes with segmentation fault on the start, specificaly, on my main.py file the line from whenever import SystemDateTime causes to segfault.

I'm already using Pydantic and give it a try because I'm using SystemDateTime on many models using Pydantic (2.11.4) and I would have to change the library otherwise.

If there is anything I can do to help debugging it, please let me know.

[ariebovenberg]

~@AllanDaemon this is probably due to an unrelated big refactor I've made. I'm happy you ran into this now because everything on CI is green 🤔 . What exact versions of your platform and python are you running?~

edit: it looks like I can reproduce. It's something that apparently only triggers when building in release mode 🤔. I will have a look.

[AllanDaemon]

~@AllanDaemon this is probably due to an unrelated big refactor I've made. I'm happy you ran into this now because everything on CI is green 🤔 . What exact versions of your platform and python are you running?~

edit: it looks like I can reproduce. It's something that apparently only triggers when building in release mode 🤔. I will have a look.

Great. I was thinking "I fell I missed something on that comment". Then eureka, context. I forgot context, but I'm glad you were able to reproduce it before I reply.

Thanks for the awesome project! And if I can be of any help with this issue, please let me know.

[ariebovenberg]

The crash is indeed unrelated, and now solved. 0.8.1 is now out and includes basic pydantic support. I'm still calling it "in preview" because it will probably need some adjustments following feedback.

[AllanDaemon]

The crash is indeed unrelated, and now solved. 0.8.1 is now out and includes basic pydantic support. I'm still calling it "in preview" because it will probably need some adjustments following feedback.

It's not crashing and it seems to be working fine now with Pydantic (I'm using only SystemDateTimethough).

Thanks a lot!

[dusktreader]

@ariebovenberg I tried out the new features in a simple FastAPI app, and it's failing for me:

  File "/home/dusktreader/git-repos/personal/space-monkey/.venv/lib/python3.13/site-packages/pydantic/json_schema.py", line 2318, in handle_invalid_for_json_schema
    raise PydanticInvalidForJsonSchema(f'Cannot generate a JsonSchema for {error_info}')
pydantic.errors.PydanticInvalidForJsonSchema: Cannot generate a JsonSchema for core_schema.PlainValidatorFunctionSchema ({'type': 'no-info', 'function': <function pydantic_schema.<locals>.<lambda> at 0x7f72c695aca0>})

For further information visit https://errors.pydantic.dev/2.11/u/invalid-for-json-schema

(full stack trace: https://gist.github.com/dusktreader/55ff6192dadd092baef92e0b39217051)

The models using whenever are very simple:

from pydantic import BaseModel
from whenever import TimeDelta


class RouteStats(BaseModel):
    requests: int = 0
    request_time: TimeDelta = TimeDelta()


class Stats(BaseModel):
    total_requests: int = 0
    total_request_time: TimeDelta = TimeDelta()
    route_stats: dict[str, RouteStats] = {}

    def update(self, route: str, span: TimeDelta):
        self.total_requests += 1
        self.total_request_time += span

        if route not in self.route_stats:
            self.route_stats[route] = RouteStats(requests=1, request_time=span)
        else:
            self.route_stats[route].requests += 1
            self.route_stats[route].request_time += span

    def reset(self):
        self.route_stats = {}
        self.total_requests = 0
        self.total_request_time = TimeDelta()


stats = Stats()

As are the endpoints that use them:

from fastapi import APIRouter
from fastapi import status

from space_monkey.config import Settings, settings
from space_monkey.schemas.info import Stats, stats

router = APIRouter(prefix="/info")


@router.get(
    "/stats",
    status_code=status.HTTP_200_OK,
    description="Endpoint used to fetch API stats",
)
async def fetch_stats() -> Stats:
    """
    Fetch API statistics
    """
    return stats


@router.get(
    "/config",
    status_code=status.HTTP_200_OK,
    description="Endpoint used to fetch the API config",
)
async def fetch_config() -> Settings:
    return settings

however, when I go to load the OpenAPI3 docs for the project, I get the failure above.

[ariebovenberg]

Yessss thanks for the quick feedback. I've managed to reduce the error to:

from pydantic import BaseModel
from whenever import TimeDelta

class Event(BaseModel):
    duration: TimeDelta

Event.model_json_schema()  # error here

A quick browse of JSON schema documentation tells me I somehow need to register whenever's types as type: string and a custom format. I'll dive in soon and fix this.

[ariebovenberg]

OK, i've managed to declare the fields as string fairly easily, ~but digging into the Pydantic code I can't find a way to pass any other keys (like format) in a way that it gets into the JSON schema.~

edit: OK I'm just a bit dense today. I'll implement __pydantic_json_schema__ 🤪

edit2: I will put out 0.8.2 to at least make JSON schema generation work. The exact content of format will require some more thought

[BayerSe]

Hey, thanks for working on this! This is what currently blocks our adoption of whenenver.

With your recent changes in pydantic_schema the following code raises TypeError instead of the expected ValidationError, as cls.parse_common_iso(v) expects a str.

@pytest.mark.parametrize("date", ["abc", 1, None, True])
def test_pydantic_raises_validation_error(date: Any) -> None:
    class Model(BaseModel):
        date: PlainDateTime

    with pytest.raises(ValidationError):
        Model(date=date)

[ariebovenberg]

Nice! This should be any easy fix as well. Will have a look later today.

[ariebovenberg]

Release 0.8.3 is now out with the change. In pydantic contexts, non-string inputs are now rejected with a ValidationError

[dusktreader]

@ariebovenberg Just tried out 0.8.3, and I'm getting a similar error when trying to open the OpenAPI3 endpoint in FastAPI:

https://gist.github.com/dusktreader/380bc15eb215391588b6faa3441b7843

[ariebovenberg]

Hmm it seems you are right. Is there any way to reproduce this error without spinning up fastAPI?—I'd like to have a proper test to ensure it keeps working in future releases.

The pydantic docs are...not as helpful as I'd like here. Only after digging into the source I find that "plain" validators are essentially incompatible with JSON schema—but then only in certain contexts...sigh

[ariebovenberg]

@dusktreader looks like you had the right idea with the "wrapping" validator. Things look to work now with FastAPI too, but it's unfortunate I can't write a test for it—I suppose a big comment will have to do...

[ariebovenberg]

Ok—I've published a prerelease 0.8.4rc0. @dusktreader can you confirm it works?