Support for PydanticV2

[Jypear]

Attempting to start work on the PydanticV2 migration.

Changes made so far:

• Temporarily comment out RedBase session attribute as it was causing issues with re-writing the value - This needs reviewing to find an alternate fix
• Converted update_forward_refs to model_rebuild in _setup.py
• set type on mac_process_count in sessions.py
• Implemented ClassVar when dealing with session variable to fix inheritence
• Convert Config Meta Class into model_config attribute
• set default values Task class as in pydanticv2 they were set to required=True as default
• _delayed_kwargs now ClassVar to be passed correctly

Pretty much all the tests still fail and this needs heavily reviewing. I'll continue to work on this PR to assist. As it stands with this version of the code a basic hello_world task can be run.

from rocketry import Rocketry

app = Rocketry()

@app.task("every 3 seconds")
def do_things():
    print("hello world")

if __name__ == "__main__":
    app.run()

This is with PydanticV2 (2.0.3) and installing ManiMozaffar red-bird branch which has quick fixes to append .v1 to the pydantic imports.

Will continue to work on making tests pass with v2 installed.

[Miksus]

Thanks a lot for making this PR! I highly appreciate it!

I'm sorry for taking such a long time to get to work on this. I felt a bit overwhelmed to get back due to the work I know what's ahead to get back to speed, and my machine doesn't run VS Code anymore so I need to upgrade that hopefully during the next week.

But maybe it's time to change my habbits.

[Miksus]

By the way, the CI crashed on:

/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/rocketry/__init__.py:1: in <module>
    from .session import Session
/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/rocketry/session.py:18: in <module>
    from rocketry.log.defaults import create_default_handler
/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/rocketry/log/defaults.py:1: in <module>
    from redbird.logging import RepoHandler
/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/redbird/__init__.py:2: in <module>
    from .base import BaseRepo, BaseResult
/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/redbird/base.py:116: in <module>
    class BaseRepo(ABC, BaseModel):
/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/redbird/base.py:1[53](https://github.com/Miksus/rocketry/actions/runs/5638564362/job/15272985407?pr=212#step:5:54): in BaseRepo
    ordered: bool = Field(default=False, const=True)
/opt/hostedtoolcache/Python/3.11.4/x[64](https://github.com/Miksus/rocketry/actions/runs/5638564362/job/15272985407?pr=212#step:5:65)/lib/python3.11/site-packages/pydantic/fields.py:723: in Field
    raise PydanticUserError('`const` is removed, use `Literal` instead', code='removed-kwargs')
E   pydantic.errors.PydanticUserError: `const` is removed, use `Literal` instead

I guess this is what you meant that we need to change Redbird as well. I'll take a look into it tomorrow (hopefully have time in the evening).

[Jypear]

Thanks a lot for making this PR! I highly appreciate it!

I'm sorry for taking such a long time to get to work on this. I felt a bit overwhelmed to get back due to the work I know what's ahead to get back to speed, and my machine doesn't run VS Code anymore so I need to upgrade that hopefully during the next week.

But maybe it's time to change my habbits.

It's no problem, I'm a big fan of this module so more than happy to try and help. You're correct the CI tests fail due to the dependancy on RedBird, I've posted the summary of the tests below so you can see where its up to when run locally. More than happy to help with a PR on RedBird as well if that would help!

I've tried my best to migrate this overall but I'm running into an issue with the last couple of tests failing due to the nature in which pydantic has changed shadowing attributes during class inheritance.

I believe its this bit of functionality here

Now for each attribute it checks to see if it exists in the base class and raises an error if it does. You can get round this by setting ClassVar, which is why you're seeing so much ClassVar as types. I'll try and take another look at it to see if there is maybe another way around this.

So apart from the last couple of tests this seems to have fixed the code enough to run, but the tests fail if you directly try to create a DummyTask that inherits Task. If you then set a variable for session in DummyTask with a ClassVar type, you get a KeyError when it tries to inherit Task and runs the validation, because session isn't passed into the validation methods doing that. (Sorry for the long winded explanation but just an update where I'm up to before I would consider this out of the draft).

So far I haven't managed to find a way around this, but I'm still trying!

Thanks so much for reviewing the changes, I'll definitely take a look at those when I get chance and push some changes.

Heres where I'm up to overall (I believe the warnings you see are for the most part depreciation warnings from RedBird. Any in Rocketry I still need to review once I have all the tests running):

================================================================================== short test summary info ===================================================================================
FAILED rocketry/test/test_build.py::test_build - AssertionError: assert '0.0.0.UNKNOWN' != '0.0.0.UNKNOWN'
FAILED rocketry/test/test_hooks.py::test_task_init - KeyError: 'session'
FAILED rocketry/test/app/test_app.py::test_task_creation - assert 3 == 5
FAILED rocketry/test/condition/test_meta.py::test_taskcond_true[main] - KeyError: 'session'
FAILED rocketry/test/condition/test_meta.py::test_taskcond_true[thread] - KeyError: 'session'
FAILED rocketry/test/condition/test_meta.py::test_taskcond_true[process] - KeyError: 'session'
FAILED rocketry/test/condition/test_meta.py::test_taskcond_false[main] - KeyError: 'session'
FAILED rocketry/test/condition/test_meta.py::test_taskcond_false[thread] - KeyError: 'session'
FAILED rocketry/test/condition/test_meta.py::test_taskcond_false[process] - KeyError: 'session'
FAILED rocketry/test/session/test_core.py::test_create - KeyError: "Task 'do_command' not found"
FAILED rocketry/test/task/test_core.py::test_defaults - KeyError: 'session'
FAILED rocketry/test/task/test_core.py::test_defaults_no_session - KeyError: 'session'
FAILED rocketry/test/task/test_core.py::test_set_timeout - KeyError: 'session'
FAILED rocketry/test/task/test_core.py::test_delete - KeyError: 'session'
FAILED rocketry/test/task/test_core.py::test_set_invalid_status - KeyError: 'session'
FAILED rocketry/test/task/test_core.py::test_failed_logging - KeyError: 'session'
FAILED rocketry/test/task/test_core.py::test_pickle - KeyError: 'session'
FAILED rocketry/test/task/test_core.py::test_crash - KeyError: 'session'
FAILED rocketry/test/task/test_core.py::test_json - KeyError: 'session'
FAILED rocketry/test/task/func/test_construct.py::test_existing_raise - KeyError: 'name'
===================================================== 20 failed, 1762 passed, 23 skipped, 8 xfailed, 3 xpassed, 17290 warnings in 41.84s =====================================================

[Miksus]

Sorry, had something unexpected yesterday and now three days of hobbies. I'll try to continue on Friday, or Saturday at the latest.

[Jypear]

Just adding a bit of an update on an issue I've been having with this. Theres one issue that's been causing me a massive headache. I'm going to be honest and say I've spent atleast 10+ hours on this issue! haha. I finally have a solution to it, but I just need your opinion on it.

Background

A couple of the tests that create a class (called DummyTask) which inherits Task have been failing. I get the following error to do with inheritance.

rocketry/test/task/test_core.py - NameError: Field name "session" shadows an attribute in parent "Task"; you might want to use a different field name with "alias='session'".

From testing this in a separate file I found no matter what I did, I could never get this type of inheritance working. I did try in dummy test setting "session" as a ClassVar to ignore this validation. It partially worked and started to initiate the task inheritance. but as soon as Task called super().__init__() it began running the pydantic validators and because session is now a ClassVar it didn't pass the argument into the validator causing a key error when trying call session from "values".

    @field_validator('name', mode="before")
    def parse_name(cls, value, values):
>       session = values.data['session']
E       KeyError: 'session'

rocketry/core/task.py:356: KeyError
_____________________________________

So the train of thought I had today was how could FuncTask inherit Task and pass session=session as an argument without raising any of these errors.

I actually ended up cloning FuncTask into a different file and tested it locally and got the inheritance error. However, as soon as I added an import to __init__.py in rocketry/tasks I was no longer getting the inheritance error.

I've looked all over the application and I can't for the life of me figure out why I stop getting the inheritance error as soon as I add an import statement in an __init__.py file anywhere inside the app.

I don't really know what I'm asking here, but have you got anything running in this application which is looking for through all import statements and doing some kind of validation? lol

The only thing remotely close is the _setup.py file. But I don't have to add the function into the forwardrefs replacement to get this to work (I only have to do that to get it to build the function correctly etc).

Proposed solution

So as stated around 19 tests are failing when trying to create a DummyTask. This isn't affecting the application itself but it only causing issues in tests.

The solution I've found is to create a new file in rocketry/tasks called _dummy.py this just has a small class for a dummy task file.

#_dummy.py
from rocketry.core import Task
class DummyTask(Task):
    def execute(self, *args, **kwargs):
        return

I then add it to the __init__.py in the same directory:

#__init__.py
from .func import FuncTask
from .code import CodeTask
from .command import CommandTask
from ._dummy import DummyTask
from . import maintain

I then need to issue model_rebuild (forward_refs) as part of the _setup.py file

...
from rocketry.tasks import CommandTask, FuncTask, CodeTask, DummyTask
...
#_setup.py
    cls_tasks = (
        Task,
        FuncTask, CommandTask, CodeTask,
        ShutDown, Restart, DummyTask,

        _FuncTaskCondWrapper
    )
    for cls_task in cls_tasks:
        #cls_task.update_forward_refs(Session=Session, BaseCondition=BaseCondition)
        cls_task.model_rebuild(
            force=True,
            _types_namespace={"Session": Session, "BaseCondition": BaseCondition}, 
            _parent_namespace_depth=4
            )

Now in the test files that used a DummyTask we replace it with an import statement. instead of putting the class in the test

# rocketry/test/task/test_core.py
...
from rocketry.tasks import DummyTask
...
def test_defaults(session):

    task = DummyTask(name="mytest", session=session)
    assert task.name == "mytest"
    assert isinstance(task.start_cond, AlwaysFalse)
    assert isinstance(task.end_cond, AlwaysFalse)

def test_defaults_no_session(session):
    with pytest.warns(UserWarning):
        task = DummyTask(name="mytest")
    assert task.session is not session
    assert isinstance(task.session, SessionClass)
    assert task.session.tasks == {task}
...
...

This then starts to pass the tests

========================================================================== 0 failed, 9 passed, 44 warnings in 0.16s ==========================================================================

So basically we are moving the dummy inside the application rather than in the tests files. This seems to be the only one I could get this to work since the change in class inheritance, and it's the last major issue I had to find a solution to. I can now start working on the rest of the deprecation warnings.

I'll push this change complete at some point tomorrow, but if you're not happy with that solution just let me know.

[Jypear]

With the solution implemented above, I have gone and ahead and took this PR out of draft. The runs fine with some brief testing I have and the test scope now passes.

Results of test:

FAILED rocketry/test/test_build.py::test_build - AssertionError: assert '0.0.0.UNKNOWN' != '0.0.0.UNKNOWN'
================================================== 1 failed, 1781 passed, 23 skipped, 8 xfailed, 3 xpassed, 16976 warnings in 41.53s ==================================================

The test_build fails because I think its some kind of weirdness with my virtualenv. The same test fails if I run the with the current main and pydantic 1.10.10 installed.

I'm pretty certain the vast majority of warnings are all coming from redbird. I couldn't find a way to exclude any warnings that come from redbird in pytest. But if I have missed something, then I would be happy to create another PR to address the rest of the deprecation warnings post redbird.

I'll keep an eye on redbirds progress, but if there isn't much shift with migrating it to v2 I'll create a PR for it.

When you get time to address some of the comments and the rest of the changes let me know and I can find alternate solutions if I have to.

[DenysMoskalenko]

Any updates here?

[Jypear]

Any updates here?

@DenysMoskalenko This PR is ready to go for the most part, apart from some depreciation warnings that may appear once RedBird is addressed. The problem is exactly that though. This project uses RedBird and the same issues with pydanticV2 exist in redbird and it also needs migrating. I've submitted a PR for RedBird as well and made a start. But I just haven't had time to work on it further due to work and stuff.

These PRs also break pydanticv1 support entirely, not sure if that would affect existing projects out there and versioning may need to be applied and supported.

Happy to give anyone access to the RedBird PR Repo if they have time to chip in.

[abinba]

Any updates?

Thanks for the initiative to add support for Pydantic 2, btw!

[Jypear]

@abinba

Any updates?

Thanks for the initiative to add support for Pydantic 2, btw!

Haven't looked at this for a bit. Last time I worked on this I was having some issues with pickleing (So some of the multiprocessing aspects of the tasks were not working (I think! some stuff worked with basic testing, but some of the tests still fail)) I had the pickleing working on an Ubuntu machine, but am having an issue on MacOS which is weird.

Single thread, async, and Multi-threaded all pass the unit tests.

Apart from that, the rest was fine in initial testing. I have also finished the PR for RedBird as well, Which is one of the requirements for Rocketry.

So I guess it's just whenever Miksus has time to get to this really. I'm pretty much ready to work on any changes that need to be made once reviewed again.

[mafrosis]

@Miksus Are you able to look at this soon? I like Rocketry, but being stuck on pydantic v1 is holding me back.

I am happy to contribute to getting this over the line, but based on comments above this PR needs your love

[zakkg3]

I normally do not do +1 posts. but I am making an exception here 🤓

[mludvig]

@Miksus any chance to proceed with this pydantic compatibility fix? Please :)