googlesearch
googlesearch copied to clipboard
Nv7-GitHub
added post method, as get was failing on me.
Changes
- Added random header, so it appears different, not just user agent.
- Added a POST version, that came from a JS free browser.
- Added a multiple search version that takes multiple search terms and returns a dictionary of urls for each.
- Added option for a random search in-between your terms as I was bulk searching similar terms and facing
too many requestsissues ( this solved that issue). - Added docstrings in all functions
- Updated requirements and readme
Extra
I think these changes add an extra option for people who just want to grab urls - it appears to be faster with the post method also and I use regex to extract the url only (currently) - working on getting the advanced descriptions etc.
not for me, I'm getting a great response, merge or no merge, i'm happy :) It's doing what I want.
On Wed, Jul 12, 2023 at 9:24 PM Nv7 @.***> wrote:
@.**** commented on this pull request.
In googlesearch/user_agents.py https://github.com/Nv7-GitHub/googlesearch/pull/57#discussion_r1261688432 :
-_useragent_list = [
- """
- return random.choice([ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0',
We only kept the common user agents since having a bunch was causing some issues.
— Reply to this email directly, view it on GitHub https://github.com/Nv7-GitHub/googlesearch/pull/57#pullrequestreview-1527178006, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIVZ7JABWDSROFYY4AAJXFDXP4BX5ANCNFSM6AAAAAAZYLMLWE . You are receiving this because you authored the thread.Message ID: @.***>
--
[image: created with MySignature.io] https://mysignature.io/?utm_source=logo Lewis Morris
M: 07554202635 E: @.*** [image: created with MySignature.io] https://github.com/lewis-morris [image: created with MySignature.io] https://stackoverflow.com/users/3348264/lewis-morris
I'm getting 429 errors - Too many requests as well. I think I can add more user agents to resolve the issue, and this PR is exactly (and above) what I need. If this is not merged, I'll raise a PR to add more user agents only.
Worked for me when I was getting the same issue.
On Thu, Sep 21, 2023 at 7:22 PM Nguyen Minh Thai @.***> wrote:
I'm getting 429 errors - Too many requests as well. I think I can add more user agents to resolve the issue, and this PR is exactly (and above) what I need. If this is not merged, I'll raise a PR to add more user agents only.
— Reply to this email directly, view it on GitHub https://github.com/Nv7-GitHub/googlesearch/pull/57#issuecomment-1730085118, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIVZ7JFQUL6SNA26GDFOENTX3SAU7ANCNFSM6AAAAAAZYLMLWE . You are receiving this because you authored the thread.Message ID: @.***>
--
[image: created with MySignature.io] https://mysignature.io/?utm_source=logo Lewis Morris
M: 07554202635 E: @.*** [image: created with MySignature.io] https://github.com/lewis-morris [image: created with MySignature.io] https://stackoverflow.com/users/3348264/lewis-morris
You could have a use_additional_agents flag which might help.
On Fri, Sep 22, 2023 at 4:40 PM Lewis Morris @.***> wrote:
Worked for me when I was getting the same issue.
On Thu, Sep 21, 2023 at 7:22 PM Nguyen Minh Thai @.***> wrote:
I'm getting 429 errors - Too many requests as well. I think I can add more user agents to resolve the issue, and this PR is exactly (and above) what I need. If this is not merged, I'll raise a PR to add more user agents only.
— Reply to this email directly, view it on GitHub https://github.com/Nv7-GitHub/googlesearch/pull/57#issuecomment-1730085118, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIVZ7JFQUL6SNA26GDFOENTX3SAU7ANCNFSM6AAAAAAZYLMLWE . You are receiving this because you authored the thread.Message ID: @.***>
--
[image: created with MySignature.io] https://mysignature.io/?utm_source=logo Lewis Morris
M: 07554202635 E: @.*** [image: created with MySignature.io] https://github.com/lewis-morris [image: created with MySignature.io] https://stackoverflow.com/users/3348264/lewis-morris
--
[image: created with MySignature.io] https://mysignature.io/?utm_source=logo Lewis Morris
M: 07554202635 E: @.*** [image: created with MySignature.io] https://github.com/lewis-morris [image: created with MySignature.io] https://stackoverflow.com/users/3348264/lewis-morris
Sorry about the long delay! I think it would be a lot cleaner if you added a new file for post requests, as the __init__.py is getting quite long. Once you do that and resolve the conflicts I can merge it. Thanks for the PR!
