Pydantic v2 support

[privatedumbo]

Making MLServer compatible with pydantic v1 AND pydantic v2.

Things added in this PR:

• removing pydantic dependency from runtimes when not required.
• redefining classes so that they use an interface compatible with versions 1 and 2 of pydantic.
• How to use with pydantic v1: install MLServer as usual, but define in the calling project an upper boundary for pydantic.
• How to use with pydantic v2: install MLServer with pip install MLServer[pydantic-v2] as an extra.

[sakoush]

@privatedumbo @franco-bocci many thanks for your contribution to provide support for pydantic v2 on MLServer. This is a something that is high on our priority list at the moment as well.

We have been discussing your change internally and we think that it might be simpler to move directly to pydantic v2 and drop support for pydantic v1 in the next release of MLServer. Our reasoning is:

• pydantic v1 EOL is June 30, 2024 as described here
• Given that pydantic EOL is very soon, we think that it will be a simpler PR and for ongoing management work from our side to move directly for pydantic v2 and drop support for v1, as otherwise we have to:
  • Add docs to explain how users can still use both versions for cases such as pip, poetry and docker images.
  • Increase test coverage to include pydantic v1 and v2 tests.
  • Remove pydantic v1 support at some point in the future.

Alternatively the plan is to release MLServer 1.6.0 with just pydantic v2 support. You can keep using MLServer 1.5.x that supports pydanic v1 for any downstream projects that still require it. We hope that this is possible in your case. Please let us know.

cc: @jesse-c @ramonpzg

[jesse-c]

follow-up: We've opened https://github.com/SeldonIO/MLServer/pull/1748 to supplant this. We really do appreciate the work in this PR as a step towards that work being possible.

[jesse-c]

We've merged in Pydantic v2 support. Again, thank you for your proposed solution and the time you spent on it! 🙇🏻