weiboSpider icon indicating copy to clipboard operation
weiboSpider copied to clipboard

Cannot crawl full post

Open chauphamcreditproduct opened this issue 5 months ago • 5 comments

First of all, thank you for sharing such a helpful method for crawling Weibo — it's been beneficial for both me and the community. However, I've encountered an issue: I'm unable to retrieve the full content of some posts, as they only display "全文" ("full text"). I would greatly appreciate your guidance on how to extract the complete content, as well as any advice or methods you could share for crawling the comments of a post.

chauphamcreditproduct avatar Jul 13 '25 20:07 chauphamcreditproduct

感谢反馈。可以修改comment_parser.py的get_long_weibo,这个方法是用来获取长微博的,因为某些原因,可能无法获取成功,为了保证效率,程序会尝试5次,如果都无法获取,就使用带有全文的内容代替。如果您想提高获取成功几率,可以把for循环中的5改成更大的整数。

dataabc avatar Jul 14 '25 08:07 dataabc

非常感谢你之前的指导,我已经顺利爬取到帖子的完整内容了!不过现在我在爬取每条帖子的评论时遇到了一些问题——我试过更换 Cookie,也调整了获取评论的相关函数代码,但依然无法成功获取评论,程序总是报错或者没有返回数据。不知道你有没有其他思路或建议可以帮我解决这个问题?

chauphamcreditproduct avatar Jul 16 '25 01:07 chauphamcreditproduct

我没研究过评论的获取,之前有网友在免cookie版(weibo-crawler)写过这个功能,不过无法确定这部分代码是否还有效。

dataabc avatar Jul 16 '25 07:07 dataabc

Thanks a lot, I appreciate your kind words, glad the code’s working well for you! 😊 Does the code support crawling by keyword, too, or is it limited to crawling by user ID only?

chauphamcreditproduct avatar Jul 21 '25 16:07 chauphamcreditproduct

本程序不支持关键词,weibo-search支持关键词,weibo-crawler支持限定用户关键词搜索。

dataabc avatar Jul 22 '25 07:07 dataabc