pytrends
pytrends copied to clipboard
GeneralMills
Is it just me or have 429 errors really increase these past few days?
I used to get 429 error every time I request more than 6 items per hour or so.
But recently, especially today I am not able to request more than 1 per hour without getting 429. Is it just my IP acting up?
Same! the only workaround I was able to do is to re-run the code multiple times in different machines or using colab with different account.
How is there no reliable solution for this in 2022 š¢ - was getting issues on pat's node js proj and was about to try this one, but then saw same thing happening
Supporting an external API for Google Trends isn't a priority for Google.
On Sat, Oct 29, 2022 at 7:27 AM, Roy Hermann < @.*** > wrote:
How is there no reliable solution for this in 2022 š¢
ā Reply to this email directly, view it on GitHub ( https://github.com/GeneralMills/pytrends/issues/538#issuecomment-1295852672 ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAIWU4PQD42HKDQLFUPJJYDWFUX63ANCNFSM6AAAAAAROMMXOI ). You are receiving this because you are subscribed to this thread. Message ID: <GeneralMills/pytrends/issues/538/1295852672 @ github. com>
Not facing the issue after using proxies and sleep 60 seconds after each request.
Not facing the issue after using proxies and sleep 60 seconds after each request.
Do you mind sharing where you're getting proxy from and how you're implementing them?
Not facing the issue after using proxies and sleep 60 seconds after each request.
Do you mind sharing where you're getting proxy from and how you're implementing them?
Iām using proxies from newipnow. Iām using 25 proxies and using random proxy on each request.
Guys, I just figured that if you downgrade the pytrend library to 4.7.2 or 4.7.3 it works. Also, collecting data for different geographical locations may stop the process, use only one location at a time, with up to 5 keywords. For more than 5 keywords, you need to apply normalization, and that is by using one shared keyword as a control in all sets of 5 keywords.
Alternatively, you may want to try R instead of Python: https://cran.r-project.org/web/packages/gtrendsR/gtrendsR.pdf This is a recent release of this month; it is more recent and reliable to use.
Guys, I just figured that if you downgrade the pytrend library to 4.7.2 or 4.7.3 it works. Also, collecting data for different geographical locations may stop the process, use only one location at a time, with up to 5 keywords. For more than 5 keywords, you need to apply normalization, and that is by using one shared keyword as a control in all sets of 5 keywords.
This worked for me! Thanks for the fix!
Would be great to figure out why downgrading solves this.
On Tue, Nov 1 2022 at 01:21, Josh Hascall < @.*** > wrote:
Guys, I just figured that if you downgrade the pytrend library to 4.7.2 or 4.7.3 it works. Also, collecting data for different geographical locations may stop the process, use only one location at a time, with up to 5 keywords. For more than 5 keywords, you need to apply normalization, and that is by using one shared keyword as a control in all sets of 5 keywords.
This worked for me! Thanks for the fix!
ā Reply to this email directly, view it on GitHub ( https://github.com/GeneralMills/pytrends/issues/538#issuecomment-1298044590 ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAIWU4LC6FE6BG5DSUQSFSDWGCSHJANCNFSM6AAAAAAROMMXOI ). You are receiving this because you commented. Message ID: <GeneralMills/pytrends/issues/538/1298044590 @ github. com>
Guys, I just figured that if you downgrade the pytrend library to 4.7.2 or 4.7.3 it works. Also, collecting data for different geographical locations may stop the process, use only one location at a time, with up to 5 keywords. For more than 5 keywords, you need to apply normalization, and that is by using one shared keyword as a control in all sets of 5 keywords.
This worked for me! Thanks for the fix!
Update: now it doesn't work... It only took 12 hours for them to block it
It is still working for me! try to increase the sleep time between requests for each subset of keywords (I use random number between 5 and 30 seconds wait). Also, don't use only one machine with the same IP address, alternate between your machine and Google Colab.
Is it working because you downgraded or because your querying strategy has changed?
On Thu, Nov 3 2022 at 00:49, Reem Omer < @.*** > wrote:
It is still working for me! try to increase the sleep time between requests for each subset of keywords (I use random number between 5 and 30 seconds wait). Also, don't use only one machine with the same IP address, alternate between your machine and Google Colab.
ā Reply to this email directly, view it on GitHub ( https://github.com/GeneralMills/pytrends/issues/538#issuecomment-1301636319 ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAIWU4MQEMHX7QEYYPRONETWGM75HANCNFSM6AAAAAAROMMXOI ). You are receiving this because you commented. Message ID: <GeneralMills/pytrends/issues/538/1301636319 @ github. com>
Both! I figured that even if I downgraded and sent too many requests, Google may block my IP, therefore, I had to change the IP this can be done by using proxies, vpn, Google Colab has also different range of IPs.
How do you know the downgrade is the cause of success? Could you help identify any changes?
On Thu, Nov 3 2022 at 03:46, Reem Omer < @.*** > wrote:
Both! I figured that even if I downgraded and sent too many requests, Google may block my IP, therefore, I had to change the IP this can be done by using proxies, vpn, Google Colab has also different range of IPs.
ā Reply to this email directly, view it on GitHub ( https://github.com/GeneralMills/pytrends/issues/538#issuecomment-1301743075 ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAIWU4KKFZKX2R5XJCTUQNTWGNUVJANCNFSM6AAAAAAROMMXOI ). You are receiving this because you commented. Message ID: <GeneralMills/pytrends/issues/538/1301743075 @ github. com>
@ReemOmer I am curious, how do you use Google Colab to scrape? I don't believe they have an API... And I haven't found any guide like that....
@emlazzarin honestly I didn't open any of the main files in both versions, so I don't know what the difference is. @igalci I used the same way of running the code in Jupyter notebook or a regular Python file run using the cmd. You will still use the Pytrends library and call all its functions.
I have an extremely novice understanding of the inner-workings of the package, but could this problem have something to do with cookies expiring on the trends.google site? I previously have been able to workaround 429 errors with this solution but now that doesn't work either. Scrolling through the request headers, I noticed a cookie expiration time that ends in the same minute as submitting the request.
Looks like no user-agent is specified in the requests, meaning they are blocked more often. I fixed it here: https://github.com/GeneralMills/pytrends/commit/18f230dd69c2cd89c203c967ac139a8218975f4d
