core
core copied to clipboard
home-assistant
Pydantic v2 migration issue
The problem
Pydantic released v2 of their library. Some libraries want to utilize the new features of Pydantic v2 so HA should update to Pydantic v2. To not burden all libraries to rework, Pydantic is providing v1 in a separate package of v2. This has been implemented in several libraries already made by some HA community members (https://github.com/AngellusMortis/pyunifiprotect/pull/297). This way the library supports both v1 and v2, this way of working is preferred to not burden the v2 upgrade with a lot of libraries to update at once.
After installing all dependencies and using pipdeptree we see that the current libraries use pydantic.
pydantic==1.10.12
├── aiolivisi==0.0.19 [requires: pydantic]
├── aionotion==2023.5.5 [requires: pydantic>=1.10.7,<2.0.0]
├── aioopenexchangerates==0.4.0 [requires: pydantic>=1.9,<2.0]
├── aiopurpleair==2022.12.1 [requires: pydantic>=1.10.2,<2.0.0]
├── aiowaqi==0.2.1 [requires: pydantic>=1.10.8]
├── demetriek==0.4.0 [requires: pydantic>=1.9.0,<2.0.0]
├── elgato==4.0.1 [requires: pydantic>=1.8.0,<2.0.0]
├── gcal-sync==4.1.4 [requires: pydantic>=1.9.0,<2.0a]
├── google-nest-sdm==2.2.5 [requires: pydantic>=1.10.4]
├── ical==5.0.1 [requires: pydantic>=1.9.1]
│ ├── gcal-sync==4.1.4 [requires: ical>=4.2.5]
│ └── pyrainbird==4.0.0 [requires: ical>=4.2.9]
├── inflect==6.0.4 [requires: pydantic>=1.9.1]
│ ├── jaraco.itertools==6.2.1 [requires: inflect]
│ │ └── jaraco.abode==3.3.0 [requires: jaraco.itertools]
│ └── jaraco.text==3.11.1 [requires: inflect]
│ ├── jaraco.collections==4.2.0 [requires: jaraco.text]
│ │ ├── jaraco.abode==3.3.0 [requires: jaraco.collections]
│ │ ├── jaraco.email==3.1.0 [requires: jaraco.collections]
│ │ │ └── jaraco.net==9.3.1 [requires: jaraco.email]
│ │ │ └── jaraco.abode==3.3.0 [requires: jaraco.net>=9]
│ │ └── jaraco.net==9.3.1 [requires: jaraco.collections]
│ │ └── jaraco.abode==3.3.0 [requires: jaraco.net>=9]
│ ├── jaraco.email==3.1.0 [requires: jaraco.text>=1.3]
│ │ └── jaraco.net==9.3.1 [requires: jaraco.email]
│ │ └── jaraco.abode==3.3.0 [requires: jaraco.net>=9]
│ └── jaraco.net==9.3.1 [requires: jaraco.text]
│ └── jaraco.abode==3.3.0 [requires: jaraco.net>=9]
├── intellifire4py==2.2.2 [requires: pydantic]
├── lacrosse-view==1.0.1 [requires: pydantic>=1.9.0]
├── open-meteo==0.2.1 [requires: pydantic>=1.8.0,<2.0.0]
├── peco==0.0.29 [requires: pydantic>=1.9.0]
├── pvo==1.0.0 [requires: pydantic>=1.8.0,<2.0.0]
├── pyaussiebb==0.0.15 [requires: pydantic>=1.9.0,<2.0.0]
├── pyrainbird==4.0.0 [requires: pydantic>=1.10.4]
├── python-bsblan==0.5.11 [requires: pydantic>=1.9.0]
├── python-kasa==0.5.3 [requires: pydantic>=1,<2]
├── python-opensky==0.2.0 [requires: pydantic>=1.10.8]
├── pytraccar==1.0.0 [requires: pydantic>=1,<2]
├── pyunifiprotect==4.10.6 [requires: pydantic!=1.9.1]
├── radios==0.1.1 [requires: pydantic>=1.9,<2.0]
├── sfrbox-api==0.0.6 [requires: pydantic>=1.10.2]
├── systembridgeconnector==3.4.9 [requires: pydantic>=1.9.0]
├── tailscale==0.2.0 [requires: pydantic>=1.8.0,<2.0.0]
├── vehicle==1.0.1 [requires: pydantic>=1.8.0,<2.0.0]
├── withings-api==2.4.0 [requires: pydantic>=1.7.2,<2.0.0]
├── xbox-webapi==2.0.11 [requires: pydantic]
├── yolink-api==0.3.0 [requires: pydantic>=1.9.0]
├── youtubeaio==1.1.5 [requires: pydantic>=1.10.8]
└── zwave-js-server-python==0.50.1 [requires: pydantic>=1.10.0]
This issue will be the progress tracker of this migration.
❌ aiolivisi
Library: [aiolivisi](https://github.com/StefanIacobLivisi/aiolivisi) Integrations using this library: [Livisi](https://github.com/home-assistant/core/tree/dev/homeassistant/components/livisi) Code owners: @StefanIacobLivisi @planbnet- [ ] Supports Pydantic v2 (PR: https://github.com/StefanIacobLivisi/aiolivisi/pull/12)
- [ ] Is updated in HA (PR: # )
Remarks:
✔️ aionotion
Library: aionotion Integrations using this library: Notion Code owners: @bachya
- [x] Supports Pydantic v2 (PR: https://github.com/bachya/aionotion/pull/251)
- [x] Is updated in HA (PR: #111388 )
Remarks: There seems to be no version that supports both v1 and v2, so this library should be directly bumped with the v2 update.
✔️ aioopenexchangerates
-- Library: aioopenexchangerates Integrations using this library: OpenExchangeRates Code owners: @MartinHjelmare
- [ ] Supports Pydantic v2 (PR: )
- [x] Is updated in HA (PR: #125593 )
Remarks:
❌ aiopurpleair
-- Library: aiopurpleair Integrations using this library: PurpleAir Code owners: @bachya
- [x] Supports Pydantic v2 (PR: https://github.com/bachya/aiopurpleair/pull/195)
- [ ] Is updated in HA (PR: # )
Remarks: There seems to be no version that supports both v1 and v2, so this library should be directly bumped with the v2 update.
✔️ aiowaqi
-- Library: aiowaqi Integrations using this library: WAQI Code owners: @joostlek
- [x] Supports Pydantic v2 (PR: Initial commit)
- [x] Is updated in HA (PR: #98000)
Remarks:
❌ demetriek
-- Library: demetriek Integrations using this library: lametric Code owners: @robbiet480 @frenck @bachya
- [ ] Supports Pydantic v2 (PR: https://github.com/frenck/python-demetriek/pull/531)
- [ ] Is updated in HA (PR: # )
Remarks:
✔️ elgato
-- Library: elgato Integrations using this library: elgato Code owners: @frenck
- [x] Supports Pydantic v2 (PR: https://github.com/frenck/python-elgato/pull/833)
- [x] Is updated in HA (PR: #102405 )
Remarks:
✔️ gcal-sync
-- Library: gcal-sync Integrations using this library: Google Calendar Code owners: @allenporter
- [x] Supports Pydantic v2 (PR: https://github.com/allenporter/gcal_sync/pull/275)
- [x] Is updated in HA (PR: #102010)
Remarks: Supports shims. Needs a release so we can bump the dependency beforehand. Latest version also has ical 5.0.1, and that has pydantic v2 support since 5.0.0.
✔️ google-nest-sdm
-- Library: google-nest-sdm Integrations using this library: Nest Code owners: @allenporter
- [x] Supports Pydantic v2 (PR: https://github.com/allenporter/python-google-nest-sdm/pull/670)
- [x] Is updated in HA (PR: #99175)
Remarks:
✔️ ical
-- Library: ical Integrations using this library: Local calendar Code owners: @allenporter
- [x] Supports Pydantic v2 (PR: https://github.com/allenporter/ical/pull/207)
- [x] Is updated in HA (PR: #98998)
Remarks:
❌ inflect
-- Library: inflect Integrations using this library: Abode Code owners: @shred86
- [ ] Supports Pydantic v2 (PR: )
- [ ] Is updated in HA (PR: # )
Remarks: This seems to be a deeper problem. I need to dive in to this to know the status.
❌ intellifire4py
-- Library: intellifire4py Integrations using this library: IntelliFire Code owners: @jeeftor
- [ ] Supports Pydantic v2 (PR: )
- [ ] Is updated in HA (PR: # )
Remarks:
❌ lacrosse-view
-- Library: lacrosse-view Integrations using this library: LaCrosse View Code owners: @IceBotYT
- [ ] Supports Pydantic v2 (PR: )
- [ ] Is updated in HA (PR: # )
Remarks:
✔️ open-meteo
-- Library: open-meteo Integrations using this library: Open Meteo Code owners: @frenck
- [x] Supports Pydantic v2 (PR: https://github.com/frenck/python-open-meteo/pull/353)
- [x] Is updated in HA (PR: #103613)
Remarks: To verify: There has been no code change to support it, but in the lock file is 2.0.3 so I am expecting this works. If this is as expected, it still needs a release and to be bumped.
❌ peco
-- Library: peco Integrations using this library: PECO Outage Counter Code owners: @IceBotYT
- [ ] Supports Pydantic v2 (PR: )
- [ ] Is updated in HA (PR: # )
Remarks: There has been no explicit PR to fix support for v2. This might work but needs to be verified.
✔️ pvo
-- Library: pvo Integrations using this library: PVOutput Code owners: @frenck
- [x] Supports Pydantic v2 (PR: https://github.com/frenck/python-pvoutput/pull/394)
- [x] Is updated in HA (PR: #102398 )
Remarks:
❌ pyaussiebb
-- Library: pyaussiebb Integrations using this library: Aussie Broadband Code owners: @nickw444 @Bre77
- [x] Supports Pydantic v2 (commit)
- [ ] Is updated in HA (PR: #99077)
Remarks: Library does not support v1 shims. Should be bumped with the v2 bump. cc @yaleman
✔️ pyrainbird
-- Library: pyrainbird Integrations using this library: Rain Bird Code owners: @konikvranik @allenporter
- [x] Supports Pydantic v2 (PR: https://github.com/allenporter/pyrainbird/pull/217)
- [x] Is updated in HA (PR: #96610)
Remarks: Still needs to update ical to =>5.0.0 as ical relies on pydantic as well.
✔️ python-bsblan
-- Library: python-bsblan Integrations using this library: BSB-Lan Code owners: @liudger
- [ ] Supports Pydantic v2 (PR: )
- [ ] Is updated in HA (PR: # )
Remarks:
✔️ python-kasa
-- Library: python-kasa Integrations using this library: TP-Link Kasa Smart Code owners: @rytilahti @thegardenmonkey
- [x] Supports Pydantic v2 (PR: https://github.com/python-kasa/python-kasa/pull/504)
- [x] Is updated in HA (PR: #103038)
Remarks:
✔️ python-opensky
-- Library: python-opensky Integrations using this library: OpenSky Network Code owners: @joostlek
- [x] Supports Pydantic v2 (PR: https://github.com/joostlek/python-opensky/pull/85)
- [x] Is updated in HA (PR: #97687)
Remarks:
✔️ pytraccar
-- Library: pytraccar Integrations using this library: Traccar Code owners: @ludeeus
- [ ] Supports Pydantic v2 (PR: )
- [x] Is updated in HA (PR: #103318)
Remarks:
✔️ pyunifiprotect
-- Library: pyunifiprotect Integrations using this library: asd Code owners: @AngellusMortis @bdraco
- [x] Supports Pydantic v2 (PR: https://github.com/AngellusMortis/pyunifiprotect/pull/297)
- [x] Is updated in HA (PR: #96486)
Remarks:
✔️ radios
-- Library: radios Integrations using this library: Radio Browser Code owners: @frenck
- [ ] Supports Pydantic v2 (PR: )
- [x] Is updated in HA (PR: #103614)
Remarks:
✔️ sfrbox-api
-- Library: sfrbox-api Integrations using this library: SFR Box Code owners: @epenet
- [ ] Supports Pydantic v2 (PR: )
- [x] Is updated in HA (PR: #125732 )
Remarks:
❌ systembridgeconnector
-- Library: systembridgeconnector Integrations using this library: System Bridge Code owners: @timmo001
- [ ] Supports Pydantic v2 (PR: )
- [ ] Is updated in HA (PR: # )
Remarks:
✔️ tailscale
-- Library: tailscale Integrations using this library: asd Code owners: @frenck
- [x] Supports Pydantic v2 (PR: https://github.com/frenck/python-tailscale/pull/370)
- [ ] Is updated in HA (PR: # )
Remarks: To verify: There has been no code change to support it, but in the lock file is 2.0.3 so I am expecting this works. If this is as expected, it still needs a release and to be bumped.
✔️ vehicle
-- Library: vehicle Integrations using this library: RDW Code owners: @frenck @joostlek
- [x] Supports Pydantic v2 (PR: https://github.com/frenck/python-vehicle/pull/441)
- [x] Is updated in HA (PR: #102379 )
Remarks:
✔️ withings-api
-- Library: withings-api Integrations using this library: Withings Code owners: @vangorra
- [ ] Supports Pydantic v2 (PR: )
- [x] Is updated in HA (PR: #101819)
Remarks: Is now/will be replaced with a new lib without pydantic
❌ xbox-webapi
-- Library: xbox-webapi Integrations using this library: Xbox Code owners: @hunterjm
- [ ] Supports Pydantic v2 (PR: )
- [ ] Is updated in HA (PR: # )
Remarks:
✔️ yolink-api
-- Library: yolink-api Integrations using this library: YoLink Code owners: @matrixd2
- [x] Supports Pydantic v2 (PR: https://github.com/YoSmart-Inc/yolink-api/commit/836e61b8b755cd180291fbea7aef824fc12c2580)
- [x] Is updated in HA (PR: #105124)
Remarks:
✔️ youtubeaio
-- Library: youtubeaio Integrations using this library: YouTube Code owners: @joostlek
- [x] Supports Pydantic v2 (PR: )
- [x] Is updated in HA (PR: # )
Remarks:
✔️ zwave-js-server-python
-- Library: zwave-js-server-python Integrations using this library: Z-Wave Code owners: @home-assistant/z-wave
- [x] Supports Pydantic v2 (PR: https://github.com/home-assistant-libs/zwave-js-server-python/pull/703)
- [x] Is updated in HA (PR: #94760)
Remarks:
--
For the core members, feel free to update where needed. For others, let us know if something has updated in a comment and we will update it.
cc @cdce8p
As also reported on Discord. But replying here as well, as I was tagged. I have no intention of supporting v2 of Pydantic in my libraries.
Instead, I might be re-evaluating the use of Pydantic overall.
../Frenck
@frenck if you do find a good alternative please do share it here. Others may want to choose the same option. I have v2 semi working but since all the internals have changed and are IMO more complicated I haven't been interested much in investing time into it
A good reminder: one can retain the V1 standards while upgrading to the V2 library. This is an excellent way to unstick this PR without committing yourself to a complete upgrade (assuming they don't get rid of this path at some point).
https://docs.pydantic.dev/latest/migration/#continue-using-pydantic-v1-features
As far as I am aware, it doesn't support everything backward. As said, I have no plans of supporting V2 at this point, and I am not willing to spend any second on it until I have explored other options.
../Frenck
pydantic has been fully removed from tailscale and elgato (and thus no longer taking part in this issue).
pydantic has been fully removed from vehicle (RDW integration) and thus no longer taking part in this issue.
Right, but those don't take part in the migration process. It simply aren't using Pydantic at all anymore.
pydantic has been fully removed from pvo (PVOutput integration) and thus no longer taking part in this issue.
pydantic has been fully removed from open-meteo and thus no longer taking part in this issue.
PR @ Core: https://github.com/home-assistant/core/pull/103613
pydantic has been fully removed from radios (Radio Browser integration) and thus no longer taking part in this issue.
PR @ Core: https://github.com/home-assistant/core/pull/103614
Man, what a liberating feeling! I should have learned about other options before using Pydantic in the first place. It would have saved me from many headaches now and in the past.
@frenck Are you using mashumaro as a replacement? I ask because I don't see a way to use it to validate a dict based on a TypedDict which is what I am looking for, so may have to keep looking :/
FYI: beartype
I found this issue, while searching why the used pydantic version is so old. Maybe it's just me, but I then found beartype which I've never heard of before. I've switched from pydantic to beartype today. I'm still astonished how simple that was and how it immediately found the bug, that already cost me several hours today.
Steps I did:
- remove every mention of pydantic
- into
__init__.pyadd:
from beartype.claw import beartype_this_package
beartype_this_package()
Sure, although I'm not sure it's worth looking: https://github.com/AlexanderLanin/homeassistant-sungrow/blob/main/custom_components/init.py
So far it found incorrect assignment to dataclasses and incorrect parameter types in functions. Both in addition to running mypy.
See https://beartype.readthedocs.io/en/latest/ for more info.
See https://beartype.readthedocs.io/en/latest/ for more info.
That looks pretty promising. Thank you for sharing!
I - and I think others - get benefit from the ser/de + coercion aspects of Pydantic which beartype doesn't do :(.
I think this worked for my codebase:
try:
from pydantic.v1 import Field, validator # type: ignore # noqa F401 # pragma: no cover
from pydantic.v1 import BaseModel # type: ignore # pragma: no cover
except ImportError:
from pydantic import Field # type: ignore # pragma: no cover
from pydantic import BaseModel # type: ignore # pragma: no cover
Pydantic will be removed from systembridgeconnector in #107957
The dependent package systembridgemodels now only uses standard dataclasses.
fyi, https://github.com/pydantic/pydantic/pull/9042 will allow cleaning up the compat code a bit when it ships.
fyi, pydantic/pydantic#9042 will allow cleaning up the compat code a bit when it ships.
This has now shipped, and is available in Pydantic 1.10.15.
fyi, pydantic/pydantic#9042 will allow cleaning up the compat code a bit when it ships.
This has now shipped, and is available in Pydantic 1.10.15.
It does not work. They also have no plans on fixing since v1 is out of support. They are working on v3 now and we still have not even gotten to v2. The Xbox integration (@hunterjm) is the only integration with any notable amount of users. Can we just consider moving to v2 or at least set a deadline for the integrations to migrate?
Owners refusing to support v2 via the shims is not fair to anyone else and it is preventing anyone from being able to use the performance improvements of v2. And once v3 comes out, it means a bunch of integrations are either going to go abandoned or those few owners are going to force everyone to rewrite their libraries just because they disagree with out a tool decided to implement something.
I want to spend more time on this when I am done with my thesis, I definetly want to speed the whole process up then
xbox seems to support v2 now so it might be able to be bumped with v2 as well
https://github.com/OpenXbox/xbox-webapi-python/commit/307a36de16ba2f0db42f5100d2560cc386e9c70d
It does not work. They also have no plans on fixing since v1 is out of support. They are working on v3 now and we still have not even gotten to v2.
Unsure what you are referring to. Are you referring to this comment?
Indeed, looks like things weren't fully implemented there. PR welcome with a fix! This is relatively low priority though, given that it's on the V1 end of things.
There is by all accounts an active effort to fix / improve the v1 shim. The comment is just saying that it's low priority, and quite rightfully.
