AssertionError (`assert 'domops' in response`) on facebook-user

[Sophiead]

Dear all,

When I use the command "snscrape --progress --jsonl facebook-user xxx > xxx.jsonl" I receive this error:

2021-03-26 10:39:18.508  CRITICAL  snscrape._cli  Dumped stack and locals to /tmp/snscrape_locals_u88a2i1j
Traceback (most recent call last):
  File "/home/xxx/.local/bin/snscrape", line 11, in <module>
    load_entry_point('snscrape==0.3.5.dev96+g47fbc2a', 'console_scripts', 'snscrape')()
  File "/home/xxx/.local/lib/python3.8/site-packages/snscrape/_cli.py", line 270, in main
    for i, item in enumerate(scraper.get_items(), start = 1):
  File "/home/xxx/.local/lib/python3.8/site-packages/snscrape/modules/facebook.py", line 194, in get_items
    assert 'domops' in response
AssertionError

I'm using snscrape version 0.3.5.dev96+g47fbc2a.

How can I solve this problem?

Thank you

[JustAnotherArchivist]

Interesting, I haven't seen that error before. It works fine for me with a few test users and from two locations. Does it happen for all users?

Facebook has been slowly and randomly rolling out the redesign over the past half year or so, so that could be one reason.

[Sophiead]

Thank you for your response! It's not for all pages the same; especially for larger pages. Now I get no error, but 0 results, or I receive a TooManyRedirects (Exceeded 30 directs) error.

[JustAnotherArchivist]

Ok, I was able to reproduce it now. The response includes "require":[["ServerRedirect","redirectPageTo",[],["\\/login\\/?next=\\u00252F",true,false]]], which indicates rate limiting/a partial IP ban. Happened on the 10th pagination request within 20 seconds for me, and afterwards scraping fails entirely. I'm not sure this can be fixed.

[nvanderperren]

A colleague and I have the same issue. She was able to get 200 posts and was blocked since then. I've got 2 posts and can't scrape the account since then.

[JustAnotherArchivist]

I triggered this in a browser as well within a minute of scrolling on one profile page. I guess that means Facebook will go the way of Instagram now, effectively forcing people to log in just to look at content. Some experimentation will be needed to find safe rate limits and the ban duration.