How to Increase the number of returned comments

[khairied]

Hi @neon-ninja , Good Morning from Algeria; Just would like to know how to set the number of retrieved comment of particular posts? I tried this for post in get_posts("zuck", options={"comments": True,"limit": 13000, "progress": True},cookies="cookies.txt" ): and this for post in get_posts("zuck", options={"comments": 13000, "progress": True}, cookies="cookies.txt" ): but the number of retrieved comments was 5000? is there a limit of facebook-scraper ? Im wondering if you can help??? How can I set a number of retrieved comments? Can I use facebook-scraper in order to collect data in real-time?? Thank you very much in advance!!!!

[neon-ninja]

Hi,

for post in get_posts("zuck", options={"comments": 13000, "progress": True}, cookies="cookies.txt" ): is the correct syntax. Yes, this limit was in place to reduce the risk of temporary bans. I've increased the limit to 1e9 in https://github.com/kevinzg/facebook-scraper/commit/3a445457e0f74995d3f332703302c874df3552c5, give that a try.

At this scale, you may want to set options={"comments": "generator"} and consume the comments at your desired pace.

Can I use facebook-scraper in order to collect data in real-time??

It depends what you mean by real-time. You can run it regularly. For example, you could run it every minute to check for new posts. So, near real-time.

[khairied]

Dear @neon-ninja ; Thank you very much for or response;

Can I use facebook-scraper in order to collect data in real-time??

I mean b this if can I retrieve the comments of particular posts in real-time?? options={"comments": "generator"} this does not retrieve the comments? This retrieves only the post

when I try this {"comments": 13000} This retrieve about 5000 comments

Thank you very much for your help!!!

[neon-ninja]

I mean b this if can I retrieve the comments of particular posts in real-time??

No, but if you check regularly enough, you can get them near real-time

options={"comments": "generator"} this does not retrieve the comments? This retrieves only the post

This returns a generator. It's something you can iterate through to retrieve comments. Like,

for comment in post["comments_full"]:
  print(comment)
  time.sleep(1)

This retrieve about 5000 comments

You would also need to update to the latest master branch, like so: pip install git+https://github.com/kevinzg/facebook-scraper.git

[MatteoSerafino]

Hi @neon-ninja when you say:

It depends what you mean by real-time. You can run it regularly. For example, you could run it every minute to check for new posts. So, near real-time.

How could we do that? As far as I can see the DateTime of comments are not in time order, so a real-time scrape would require a complete scrape and a check for new comments by means of their ID. Am I wrong?

Of course, I refer to posts younger than 24 hours.

[neon-ninja]

You're not wrong, this is a limitation of m.facebook.com. Perhaps you could check the total comment count, and only extract comments if it's increased significantly.