google-group-crawler icon indicating copy to clipboard operation
google-group-crawler copied to clipboard

Nothing happen

Open RobbiNespu opened this issue 3 years ago • 2 comments

Hi, I tried cloned it and execute as instructed

$ pwd
/home/robbi/workplace/bash/google-group-crawler

$ export _GROUP="debmal"

$ ./crawler.sh -sh

$ ./crawler.sh -sh > curl.sh

$ /usr/bin/bash +x curl.sh

but nothing happen, inside debmal threads, mbox also empty. Maybe my steps are wrong or maybe regex on script are broken. Could you have a look on it?

Thanks for the great works!

RobbiNespu avatar Jul 29 '21 10:07 RobbiNespu

@RobbiNespu I've just had a quick look. I believe that Google has changed [recently] their front-end application, and yeah, all the old ajax support (?_escaped_fragment_ part in URI) have just gone. When that happens, the script has no way to get any data from Google!

I will try again if there is any other work-around. You know, google promises... nothing. That's very sad :(

icy avatar Jul 29 '21 10:07 icy

  • Proposal (2009): https://developers.google.com/search/blog/2009/10/proposal-for-making-ajax-crawlable
  • Deprecation (2015): https://developers.google.com/search/blog/2015/10/deprecating-our-ajax-crawling-scheme

icy avatar Jul 29 '21 10:07 icy