snscrape icon indicating copy to clipboard operation
snscrape copied to clipboard

`KeyError: 'mp4_720p_mp4'` on Weibo scrapes

Open aizande opened this issue 2 years ago • 1 comments

When I attempt the following call (or any other weibo user)

snscrape --jsonl weibo-user 1223717857  

I receive the following error:

2022-07-06 09:04:36.550  CRITICAL  snscrape._cli  Dumped stack and locals to /tmp/snscrape_locals_9a6n2i1t
Traceback (most recent call last):
  File "/home/user/.local/bin/snscrape", line 8, in <module>
    sys.exit(main())
  File "/home/user/.local/lib/python3.8/site-packages/snscrape/_cli.py", line 318, in main
    for i, item in enumerate(scraper.get_items(), start = 1):
  File "/home/user/.local/lib/python3.8/site-packages/snscrape/modules/weibo.py", line 115, in get_items
    yield self._mblog_to_item(card['mblog'])
  File "/home/user/.local/lib/python3.8/site-packages/snscrape/modules/weibo.py", line 97, in _mblog_to_item
    repostedPost = self._mblog_to_item(mblog['retweeted_status']) if 'retweeted_status' in mblog else None,
  File "/home/user/.local/lib/python3.8/site-packages/snscrape/modules/weibo.py", line 95, in _mblog_to_item
    video = mblog['page_info']['media_info']['mp4_720p_mp4'] if 'page_info' in mblog and mblog['page_info']['type'] == 'video' else None,
KeyError: 'mp4_720p_mp4'

Has anyone had any success scraping weibo lately?

aizande avatar Jul 06 '22 13:07 aizande

Confirmed, thanks, I guess they changed the way videos are delivered.

JustAnotherArchivist avatar Jul 06 '22 21:07 JustAnotherArchivist