Cannot crawl full post
First of all, thank you for sharing such a helpful method for crawling Weibo — it's been beneficial for both me and the community. However, I've encountered an issue: I'm unable to retrieve the full content of some posts, as they only display "全文" ("full text"). I would greatly appreciate your guidance on how to extract the complete content, as well as any advice or methods you could share for crawling the comments of a post.
感谢反馈。可以修改comment_parser.py的get_long_weibo,这个方法是用来获取长微博的,因为某些原因,可能无法获取成功,为了保证效率,程序会尝试5次,如果都无法获取,就使用带有全文的内容代替。如果您想提高获取成功几率,可以把for循环中的5改成更大的整数。
非常感谢你之前的指导,我已经顺利爬取到帖子的完整内容了!不过现在我在爬取每条帖子的评论时遇到了一些问题——我试过更换 Cookie,也调整了获取评论的相关函数代码,但依然无法成功获取评论,程序总是报错或者没有返回数据。不知道你有没有其他思路或建议可以帮我解决这个问题?
我没研究过评论的获取,之前有网友在免cookie版(weibo-crawler)写过这个功能,不过无法确定这部分代码是否还有效。
Thanks a lot, I appreciate your kind words, glad the code’s working well for you! 😊 Does the code support crawling by keyword, too, or is it limited to crawling by user ID only?
本程序不支持关键词,weibo-search支持关键词,weibo-crawler支持限定用户关键词搜索。