Google Block HTTP Error 503: Service Unavailable

[limonkufu]

After 20-30 papers Google blocked the service with error on title probably due to scholar API limits. I looked it up and I saw enabling cookies could prevent this from happening.

[smidm]

The Google Scholar has no api. The data is scraped by scholar.py and the cookies are already enabled. I got no idea how to avoid being blocked by the Scholar's anti scraping mechanisms.

[limonkufu]

In the scholar.py there is this line I saw which enables cookies. I think this line needs to be set to some text file

    # If set, we will use this file to read/save cookies to enable
    # cookie use across sessions.
COOKIE_JAR_FILE = None

[limonkufu]

scholar.py supports cookies when called from CLI. We also have to use this command to save cookies. At first start I query a article and save cookies like below:

        query.set_phrase("quantum theory")
        scholar.send_query(query)
        scholar.save_cookies()