Fix 429 errors (Fixes #354, #413, #492, #523, #535, #538) and fix pandas `frame.append` deprecation warning (Fixes #525)

[xenova]

The today_searches method results in a FutureWarning from pandas:

.../pytrends/request.py:447: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.

This warning is present in the example you provide too (and has been previously reported in #525).

[xenova]

I also added a commit to fix an issue where users cannot specify headers in requests_args. This allows for a user-agent to be specified, which alleviates 429 errors.

[xenova]

Example usage:

from pytrends.request import TrendReq

requests_args = {
    'headers': {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36'
    }
}

# Only need to run this once, the rest of requests will use the same session.
pytrend = TrendReq(requests_args=requests_args)

# Continue normally
...

[gilbertovilarunc]

@xenova I am still having the 429 issue, even passing your User-agent. Should I use another "User-agent" definition? Does it depende on my current region?

[xenova]

@gilbertovilarunc Changing the user agent might help, but if you are still getting 429 issues with a custom user-agent, you are most likely getting blocked due to an excessive number of requests. VPNs and proxies might be the only solution if nothing else fixes it.

[zmotinstitute]

Hi guys,

I've started to have this 429 error since last week, just after google trends UI update on March 8th 2023.

The custom header should fix it?

I've tried to use it, but the pytrends.build_payload keeps returning the TooManyRequestsError: The request failed: Google returned a response with code 429

Here is a sneak peek o the code:

Create a "request args" to avoid 429 errors from google

requests_args = { 'headers': { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36' } }

Create a pytrends object

pytrends = TrendReq(requests_args=requests_args)

Define the keyword, date range and geolocation

keyword = keyword_to_scrape date_range = date_range_to_scrape geo = 'BR'

Which Google property to filter data from (leave empty for search, and we will always use it)

gprop = ''

Build the payload

pytrends.build_payload(kw_list=[keyword], timeframe=date_range, geo=geo, gprop=gprop)

Get the interest over time data

df = pytrends.interest_over_time()

[gubiithefish]

@zmotinstitute, I had the samme issue where using a User-Agent doesn't seem to resolve the "TooManyRequestsError" issue anymore. After tinkering about I found a solution that worked for me. Here are the steps I took to make it work.

1. Go to https://trends.google.com/trends and perform any search you like.
2. Right-click on the page and select "Inspect" to open the Developer Tools. Click on the "Network" tab.
3. Locate a GET request to the trends.google.com domain and click on it.
4. Find the REQUEST headers and copy the entire set of headers. Here's an example of what mine looked like:

   {
     "Host": "trends.google.com",
     "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/111.0",
     "Accept": "application/json, text/plain, */*",
     "Accept-Language": "en-US,en;q=0.5",
     "Accept-Encoding": "gzip, deflate, br",
     "Alt-Used": "trends.google.com",
     "Connection": "keep-alive",
     "Referer": "**_Redacted_**",
     "Cookie": "**_Redacted_**",
     "Sec-Fetch-Dest": "empty",
     "Sec-Fetch-Mode": "cors",
     "Sec-Fetch-Site": "same-origin",
     "TE": "trailers"
   }
   

5. Insert all the values into your "headers" dictionary for the request_args variable. Make sure to replace the "Referer" and "Cookie" value with your own from the headers you copied in step 4.

This method worked for me, and I hope it helps you too! 🚀

[naderabdalghani]

@gubiithefish This works for me too. I think passing a cookie in the request headers is the most crucial part, but since that cookies expire, do you know how to generate the "Cookie" value programmatically?

[naderabdalghani]

@gubiithefish This works for me too. I think passing a cookie in the request headers is the most crucial part, but since that cookies expire, do you know how to generate the "Cookie" value programmatically?

Found a solution for that, at least one that works for me. Cookies can be set using self.GetGoogleCookie(), so by adding the following line in _get_data(...) under line 131 in request.py, it should work

self.cookies = self.GetGoogleCookie()

If it still doesn't work as in my case, change every instance of the URL in self.GetGoogleCookie() from https://trends.google.com/trends/?geo= to https://trends.google.com/trends/explore/?geo= (Source)

[Raidus]

Anyone experiencing this 429 again?

[comsaint]

Anyone experiencing this 429 again?

@Raidus encounter this error today. Upgraded to 4.9.2 and tried all the above suggestions, but hit 429 in 1st request.

[valentingarnier]

Is there any fix for this 429 error so far? I've tried all of the solutions suggested but still having an issue. Running the code from GoogleColab. Thanks a lot!

[wmotkowska-stud-412081]

Hi, I use the gtrendsR in R and fixed the 429 error after setting user and password in function setHandleParameters offered by the library. Documentation: https://cran.r-project.org/web/packages/gtrendsR/gtrendsR.pdf

[njalali63]

@wmotkowska-stud-412081 Can you give more detail on what did you pass into the setHandleParameters function? The package manual was not clear on that.

[HeyTimon]

still error here on 27 march 2024, anybody have clue for 429 error pytrends?