podimo icon indicating copy to clipboard operation
podimo copied to clipboard
verified publisher icon ThijsRay

Error while fetching podcasts, but urls are found

Open stephanvierkant opened this issue 1 year ago • 4 comments

I've set up this great tool on my NAS and it worked great, but now I'm getting this error (debug=true):

ERROR | 2024-07-17T07:08:07Z | Error while fetching podcasts: 
DEBUG | 2024-07-17T07:08:07Z | Incoming GET request for 'http://[redacted]' from User-Agent okhttp/4.11.0 at 172.19.0.1.
DEBUG | 2024-07-17T07:08:07Z | Incoming GET request for 'http://[redacted]' from User-Agent okhttp/4.11.0 at 172.19.0.1.
DEBUG | 2024-07-17T07:08:07Z | Feed request for podcast 99aa420b-14d0-4ffc-8e79-a55ed8f793e4 from IP 172.19.0.1 with User-Agent:okhttp/4.11.0.
DEBUG | 2024-07-17T07:08:07Z | Got podcast 'POM - Een podcast over media, cultuur, technologie en ondernemen' (99aa420b-14d0-4ffc-8e79-a55ed8f793e4) from cache (21164 seconds left)
DEBUG | 2024-07-17T07:08:07Z | HEAD request to https://cdn.podimo.com/audios/[redacted].mp3

I've tried both Zenrows and ScraperAPI, but that doesn't work either. While it gives an error, it can found the (correct) mp3-urls. Any idea how I can fix this?

stephanvierkant avatar Jul 17 '24 07:07 stephanvierkant

Additional info:

DEBUG | 2024-07-19T14:23:37Z | Incoming GET request for 'http://127.0.0.1:12104/feed/99aa420b-14d0-4ffc-8e79-a55ed8f793e4.xml?b11b9842b3&region=nl&locale=nl-NL' from User-Agent Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 at 172.25.0.1.
2024-07-19T14:23:41.360121238Z DEBUG | 2024-07-19T14:23:41Z | Feed request for podcast 99aa420b-14d0-4ffc-8e79-a55ed8f793e4 from IP 172.25.0.1 with User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36.
2024-07-19T14:23:41.409765005Z DEBUG | 2024-07-19T14:23:41Z | AuthorizationPreregisterUser
2024-07-19T14:23:41.411415641Z DEBUG | 2024-07-19T14:23:41Z | Starting new HTTPS connection (1): podimo.com:443
2024-07-19T14:23:42.058279806Z DEBUG | 2024-07-19T14:23:42Z | https://podimo.com:443 "POST /graphql HTTP/1.1" 200 377
2024-07-19T14:23:42.059671915Z DEBUG | 2024-07-19T14:23:42Z | OnboardingQuery
2024-07-19T14:23:42.144538491Z DEBUG | 2024-07-19T14:23:42Z | https://podimo.com:443 "POST /graphql HTTP/1.1" 200 101
2024-07-19T14:23:42.145899230Z DEBUG | 2024-07-19T14:23:42Z | AuthorizationAuthorize user: "[redacted]"
2024-07-19T14:23:42.342039837Z DEBUG | 2024-07-19T14:23:42Z | https://podimo.com:443 "POST /graphql HTTP/1.1" 200 247
2024-07-19T14:23:42.343862227Z ERROR | 2024-07-19T14:23:42Z | An error occurred: Podimo returned no valid data for query query AuthorizationAuthorize($
2024-07-19T14:23:42.343908753Z Traceback (most recent call last):
2024-07-19T14:23:42.343917239Z   File "/src/main.py", line 106, in check_auth
2024-07-19T14:23:42.343923489Z     await client.podimoLogin(scraper)
2024-07-19T14:23:42.343929210Z   File "/src/podimo/client.py", line 145, in podimoLogin
2024-07-19T14:23:42.343935441Z     result = await self.post(headers, query, variables, scraper)
2024-07-19T14:23:42.343941018Z   File "/src/podimo/client.py", line 71, in post
2024-07-19T14:23:42.343947342Z     raise RuntimeError(f"Podimo returned no valid data for query {query.strip()[:30]}")
2024-07-19T14:23:42.343953723Z RuntimeError: Podimo returned no valid data for query query AuthorizationAuthorize($
2024-07-19T14:23:42.345237617Z DEBUG | 2024-07-19T14:23:42Z | Incoming GET request for 'http://127.0.0.1:12104/feed/99aa420b-14d0-4ffc-8e79-a55ed8f793e4.xml?b11b9842b3&region=nl&locale=nl-NL' from User-Agent Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 at 172.25.0.1.

stephanvierkant avatar Jul 19 '24 14:07 stephanvierkant

Ik heb nu hetzelfde probleem na het overzetten van de Podimo instance naar een nieuwe Raspberry. Exact zelfde instellingen als voor de verhuizing.

podimo  | DEBUG | 2024-07-27T07:49:09Z | Incoming POST request for 'http://podimo.[redacted]/' from User-Agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 at 172.18.0.1.
podimo  | DEBUG | 2024-07-27T07:49:28Z | Feed request for podcast 99aa420b-14d0-4ffc-8e79-a55ed8f793e4 from IP 172.18.0.1 with User-Agent:PocketCasts/1.0 (Pocket Casts Feed Parser; +http://pocketcasts.com/).
podimo  | DEBUG | 2024-07-27T07:49:28Z | AuthorizationPreregisterUser
podimo  | DEBUG | 2024-07-27T07:49:28Z | Starting new HTTPS connection (1): podimo.com:443
podimo  | DEBUG | 2024-07-27T07:49:32Z | https://podimo.com:443 "POST /graphql HTTP/1.1" 200 None
podimo  | DEBUG | 2024-07-27T07:49:32Z | OnboardingQuery
podimo  | DEBUG | 2024-07-27T07:49:32Z | https://podimo.com:443 "POST /graphql HTTP/1.1" 200 None
podimo  | DEBUG | 2024-07-27T07:49:32Z | AuthorizationAuthorize user: [redacted]
podimo  | DEBUG | 2024-07-27T07:49:32Z | https://podimo.com:443 "POST /graphql HTTP/1.1" 200 None
podimo  | DEBUG | 2024-07-27T07:49:33Z | Got podcast 'POM - Een podcast over media, cultuur, technologie en ondernemen' (99aa420b-14d0-4ffc-8e79-a55ed8f793e4) from cache (20995 seconds left)
podimo  | DEBUG | 2024-07-27T07:49:33Z | HEAD request to https://cdn.podimo.com/audios/28b52427-2c14-44fd-bf67-82088c3f4ff6.mp3
podimo  | DEBUG | 2024-07-27T07:49:33Z | HEAD request to https://cdn.podimo.com/audios/61153414-a8f5-4dee-9290-a0fd725fc429.mp3
podimo  | DEBUG | 2024-07-27T07:49:33Z | HEAD request to https://cdn.podimo.com/audios/340e2972-ff19-49ba-aa5d-5da520886916.mp3
podimo  | DEBUG | 2024-07-27T07:49:33Z | HEAD request to https://cdn.podimo.com/audios/ba609edb-137d-46fd-8aa8-6c81a2ffa175.mp3
podimo  | DEBUG | 2024-07-27T07:49:33Z | HEAD request to https://cdn.podimo.com/audios/b992a6bb-e32e-4630-ac4c-cd44174b7395.mp3
podimo  | ERROR | 2024-07-27T07:49:36Z | Error while fetching podcasts:
podimo  | DEBUG | 2024-07-27T07:49:36Z | Incoming GET request for 'http://podimo.[redacted]/feed/99aa420b-14d0-4ffc-8e79-a55ed8f793e4.xml?[redacted]&amp&region=nl&amp&locale=nl-NL' from User-Agent PocketCasts/1.0 (Pocket Casts Feed Parser; +http://pocketcasts.com/) at 172.18.0.1.

sauerhosen avatar Jul 27 '24 07:07 sauerhosen

Ik heb de try/except in main.py on line 230 uitge-comment om de onderliggende fout te zien:

podimo  | DEBUG | 2024-07-27T08:17:43Z | Got podcast 'POM - Een podcast over media, cultuur, technologie en ondernemen' (99aa420b-14d0-4ffc-8e79-a55ed8f793e4) from cache (18115 seconds left)
podimo  | ERROR | 2024-07-27T08:17:43Z | Exception on request GET /feed/99aa420b-14d0-4ffc-8e79-a55ed8f793e4.xml
podimo  | Traceback (most recent call last):
podimo  |   File "/usr/local/lib/python3.10/site-packages/quart/app.py", line 1650, in handle_request
podimo  |     return await self.full_dispatch_request(request_context)
podimo  |   File "/usr/local/lib/python3.10/site-packages/quart/app.py", line 1675, in full_dispatch_request
podimo  |     result = await self.handle_user_exception(error)
podimo  |   File "/usr/local/lib/python3.10/site-packages/quart/app.py", line 1107, in handle_user_exception
podimo  |     raise error
podimo  |   File "/usr/local/lib/python3.10/site-packages/quart/app.py", line 1673, in full_dispatch_request
podimo  |     result = await self.dispatch_request(request_context)
podimo  |   File "/usr/local/lib/python3.10/site-packages/quart/app.py", line 1718, in dispatch_request
podimo  |     return await self.ensure_async(handler)(**request_.view_args)
podimo  |   File "/src/main.py", line 187, in serve_basic_auth_feed
podimo  |     return await serve_feed(username, auth.password, podcast_id, region, locale)
podimo  |   File "/src/main.py", line 232, in serve_feed
podimo  |     podcasts = await podcastsToRss(
podimo  |   File "/src/main.py", line 346, in podcastsToRss
podimo  |     await asyncio.gather(
podimo  |   File "/src/main.py", line 290, in addFeedEntry
podimo  |     fe.pubDate(episode["publishDatetime"])
podimo  | KeyError: 'publishDatetime'

Is er wellicht iets veranderd in de Podimo response, geen "publishDatetime"? @ThijsRay, enig idee hoe ik dit verder kan onderzoeken/oplossen?

sauerhosen avatar Jul 27 '24 08:07 sauerhosen

Sorry voor spammen, maar ik heb ontdekt dat de ik soms wel, en soms niet "publishDatetime" terug krijg. logging.debug(episode) in main.py op line 287. Ik zal een patch schrijven en testen.

Op mijn raspberry: podimo | DEBUG | 2024-07-27T09:19:59Z | {'id': '9bf3befb-856e-40fc-879e-31c5352f4719', 'artist': 'POM - Een podcast over media, cultuur, technologie en ondernemen', 'podcastName': 'POM - Een podcast over media, cultuur, technologie en ondernemen', 'imageUrl': 'https://cdn.podimo.com/images/264270c1-353b-45bb-86f1-a351362d1796_400x400.png', 'description': '🧡 Hier word je lid van de Telegram community https://pom.community\n🚀 Kijk clips via onze Instagram: https://instagram.com/podcastovermedia\n💚 Volg de nieuwsbrief van Ernst-Jan: https://pfauth.substack.com \n💫 Volg de nieuwsbrief van Alexander: https://alexanderklopping.substack.com', 'datetime': '2024-07-11T04:31:21.050Z', 'title': 'Temu maakt jou kapot | POM S09E70', 'audio': {'url': 'https://cdn.podimo.com/audios/b992a6bb-e32e-4630-ac4c-cd44174b7395.mp3', 'duration': 3434}, 'streamMedia': {'duration': 3434, 'url': 'https://cdn.podimo.com/hls-media/b992a6bb-e32e-4630-ac4c-cd44174b7395/main.m3u8'}}

Op mijn development machine: DEBUG | 2024-07-27T08:54:10Z | {'id': '82600de6-e994-4a07-b70a-1ed874fa22cf', 'artist': 'POM - Een podcast over media, cultuur, technologie en ondernemen', 'podcastName': 'POM - Een podcast over media, cultuur, technologie en ondernemen', 'imageUrl': 'https://cdn.podimo.com/images/90e9aa78-f2dd-42fa-b4f5-c13e93753aea_400x400.png', 'description': "Alexander Klöpping (Blendle) en Ernst-Jan Pfauth (De Correspondent) hebben besloten u op te zadelen met de verhalen die ze normaal in de kroeg vertellen. Ze bespreken hoe Facebook Instant Articles de ondergang van kranten en tijdschriften inluidt, of het zin heeft om je daar tegen te verzetten, dat je 'meme' op z'n Fries uitspreekt en dat The New York Times de journalistiek wil redden met een receptenservice. Sterkte!\nShownotes: https://pfauth.com/nieuwe-media/podcast#bronnen\n\nZie het privacybeleid op https://art19.com/privacy en de privacyverklaring van Californië op https://art19.com/privacy#do-not-sell-my-info.", 'datetime': '2016-05-11T22:33:26.000Z', 'publishDatetime': '2016-05-11T22:33:26.000Z', 'title': "S01E01 - 'Facebook gaat alles kapot maken'", 'audio': {'url': 'https://cdn.podimo.com/audios/6f587ef4-0221-4082-91e2-2bb885b1803b.mp3', 'duration': 3620}, 'streamMedia': {'duration': 3620, 'url': 'https://cdn.podimo.com/audios/6f587ef4-0221-4082-91e2-2bb885b1803b.mp3'}}

sauerhosen avatar Jul 27 '24 09:07 sauerhosen

Patch has been merged in https://github.com/ThijsRay/podimo/pull/46! Closing this issue.

ThijsRay avatar Aug 24 '24 12:08 ThijsRay