[BUG] TikTok seems to have changed things
Describe the bug
TikTok got rid of #SIGI_STATE and replaced it with #__UNIVERSAL_DATA_FOR_REHYDRATION__, now the data exists in ex /__DEFAULT_SCOPE__/webapp.video-detail/itemInfo and has different keys so the mapper doesn't work anymore.
To Reproduce Steps to reproduce the behavior:
- Scrape any page. Timeout occurs because SIGI_STATE isn't attached.
Expected behavior Well, you know...
Version Information
Please include what versions of pydantic, playwright, and tiktokapipy you have installed (can be found with pip freeze).
playwright==1.30.0
playwright-stealth==1.0.5
pydantic==2.5.2
pydantic_core==2.14.5
tiktokapipy==0.2.4.post2
System Information Please provide information about the system you're running the library on. What's the OS? Is it a desktop, server, VM, or container? Various linux (ubuntu/debian) machines Region Information Where in the world are you located? If you're not in the US, it's likely that TikTok treats your region differently, which may present some challenges. US Additional context Add any other context about the problem here.
As of today, i only see static and user data in the new state object, eg __$UNIVERSAL_DATA$__.__DEFAULT_SCOPE__["webapp.user-detail"] but no webapp.video-detail. Video data should be gathered through the DeferredItemListIterator(self._api, "post", self.sec_uid) magic anyways, so shmaybe not a big change needed?
I'll need some time to address this. At the moment, I'm pretty busy with research and various applications, but I'll try to get to this soon.