Update to pydantic v2

[dragosmc]

Is your feature request related to a problem? Please describe. Pydantic v2 has been out for a few months now and it brings new cool features as well as big performance improvements due to its use of Rust for validation. Currently Ludwig requires Pydantic < 2.0.

The conflict is caused by:
    The user requested pydantic==2.4.2
    ludwig 0.8.4 depends on pydantic<2.0

Are there any plans to update to the new version?

I might be able to help with this.

[justinxzhao]

Hi @dragosmc,

We had to pin pydantic to <2.0 in https://github.com/ludwig-ai/ludwig/pull/3537 because it was incompatible with transformers.

If we could get your help to check if transformers is compatible, and then releasing the pin on Ludwig, that would be great!

[dragosmc]

I see. In that case I guess we'll wait for them to update (or one of their dependencies) as they are still forcing pydantic<2 at the moment. Thanks.

[dragosmc]

Hi, I'm reopening this as transformers have updated their code to use pydantic2 https://github.com/huggingface/transformers/issues/27933 and it feels like ludwig could remove the version restriction.