Cannot Find Images for this Search Filter

[vk379]

Hey, I recently noticed, google_images_download is failing to download images at random times. An example of this would be, if I made a loop to download a certain image, and tried to download it 10 times, it would download it most of the time, but fail at random points. For example, iteration 1 of the loop successfully downloads the images, but iteration 9 fails , for no reason, even though the arguments are kept constant. Could use any and all help on this issue. The Error I receive is: Unfortunately all 50 could not be downloaded because some images were not downloadable. 0 is all we got for this search filter!

Thanks, -vk379

[S-Cardenas]

I have been seeing this issue also and would love to know what's behind it. Could it be that Google is prevent you from downloading images if it detects the same IP address being used too quickly?

[vk379]

I'm not entirely sure. I fixed the issue personally but my code is really inefficient. I am basically running a while loop until till it works.. which means I am calling google images around 3-4 times till I get the URLs I want. Pretty brute force but does the job, would love to see a solution for this issue though.

[S-Cardenas]

I was thinking of doing the same thing but don't want to brute force it. I'm more worried about the root of the issue being Google related.

[S-Cardenas]

After some digging around, I found the issue to be the result of Google updating the response format. Traditionally, each of Google's image result came in the form of:

<div jscontroller="Q7Rsec" data-ri="0" class="rg_bx rg_di rg_el ivg-i" data-ved="0ahUKEwj0zuvFn_TlAhXJc98KHQC0CAcQMwhkKAAwAA">
    <a jsname="hSRGPd" href="#" jsaction="fire.ivg_o;mouseover:str.hmov;mouseout:str.hmou" class="rg_l" rel="noopener">
        <div class="THL2l"></div><img id="CixhSoPkCojjCM:" src="data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" jsaction="load:str.tbn" class="rg_ic rg_i" alt="Image result for economy chart" data-deferred="1">
        <div class="rg_ilmbg"> 2161&nbsp;&#215;&nbsp;1910 </div>
    </a>
    <a class="iKjWAf irc-nic isr-rtc a-no-hover-decoration" href="#" jsaction="mouseover:m8Yy5c;mousedown:QEDpme;focus:QEDpme;" rel="noopener" target="_blank">
        <div class="mVDMnf nJGrxf">The $80 Trillion World Economy in One Chart</div>
        <div class="nJGrxf FnqxG"><span>visualcapitalist.com</span></div>
    </a>
    <div class="rg_meta notranslate">{"id":"CixhSoPkCojjCM:","isu":"visualcapitalist.com","itg":0,"ity":"jpg","oh":1910,"ou":"http://2oqz471sa19h3vbwa53m33yj-wpengine.netdna-ssl.com/wp-content/uploads/2018/10/world-economy-gdp.jpg","ow":2161,"pt":"The $80 Trillion World Economy in One Chart","rh":"visualcapitalist.com","rid":"vzfo7BtwQ7sOEM","rmt":0,"rt":0,"ru":"https://www.visualcapitalist.com/80-trillion-world-economy-one-chart/","sc":1,"st":"Visual Capitalist","th":211,"tu":"https://encrypted-tbn0.gstatic.com/images?q\u003dtbn:ANd9GcRxpTvHqGYeKsCQATZP0ChgnXw2b4PAzSyBWHkpYNfFE1oqrDi7kg\u0026s","tw":239}</div>
    <div class="ll0QOb"></div>
</div>

This library, google-images-download, then searches for each original image link by searching for rg_meta notranslate in the body of the response. This can be seen in the method _get_next_item of google_images_download.py.

However, Google will occasionally send the response in a different format, such as this:

<div jsaction="IE7JUb:e5gl8b;MW7oTe:fL5Ibf;dtRDof:s370ud;R3mad:ZCNXMe;v03O1c:cJhY7b;" data-ved="2ahUKEwjbwvWwovTlAhWJNN8KHdolChQQMygAegUIARD_AQ" data-ictx="1" data-id="CixhSoPkCojjCM" jsname="N9Xkfe" data-ri="0" class="isv-r PNCib MSM1fd BUooTd" jscontroller="SI4J6c" jsmodel="uZbpBf sB4qxc" jsdata="j0Opre;CixhSoPkCojjCM;7" style="width:179px;" data-tbnid="CixhSoPkCojjCM" data-ct="0" data-cb="0" data-cl="0" data-cr="0" data-tw="239" data-ow="2161" data-oh="1910">
    <a class="wXeWr islib nfEiy mM5pbd" jsname="sTFXNd" jsaction="click:J9iaEb;" jsaction="mousedown:npT2md; touchstart:npT2md;" data-nav="1" tabindex="0" style="height:158px;">
        <div class="bRMDJf islir" jsname="DeysSe" style="background:rgb(248,69,133);width:179px; height:158px;" jsaction="mousedown:npT2md; touchstart:npT2md;"><img class="rg_i Q4LuWd tx8vtf" src="data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" data-iid="12" data-iurl="https://encrypted-tbn0.gstatic.com/images?q=tbn%3AANd9GcRbMq0HH9-11adBbwZ2LP9hFwRpBoe6UbFGgUvr2zNe0O5dhxyp" jsname="Q4LuWd" alt="Image result for economy chart" /></div>
        <div class="c7cjWc"></div>
        <div class="h312td  RtIwE" jsname="bOERI">
            <div class="O1vY7"><span>2161 × 1910</span></div>
        </div>
        <div class="PiLIec" jsaction="click: gFs2Re"></div>
    </a>
    <a class="VFACy kGQAp" data-ved="2ahUKEwjbwvWwovTlAhWJNN8KHdolChQQr4kDegUIARCAAg" jsname="uy6ald" rel="noopener" target="_blank" href="https://www.visualcapitalist.com/80-trillion-world-economy-one-chart/" jsaction="focus:kvVbVb; mousedown:kvVbVb; touchstart:kvVbVb;">
        <div class="sMi44c lNHeqe">
            <div class="WGvvNb">The $80 Trillion World Economy in One Chart</div>
            <div class="fxgdke">visualcapitalist.com</div>
        </div>
    </a>
</div>

This type of response can not be parsed correctly by the library because the library is specifically looking for rg_meta notranslate to find the original image link. In this updated response, the rg_meta notranslate tag and original image link are absent from the element. The original image link is being provided in a javascript callback AF_initDataCallback.

It's not clear why Google is sending two differently formatted responses. It's also not clear if they will continue to do so in the feature. This particular "bug" could be fixed via pull request but we the community should be mindful of any future changes Google implements to the response format. @vk379 @hardikvasa

[vk379]

Wow this is an amazing discovery. This makes sense, as to why I am getting so many errors. I think recently google themselves have unrolled a lot of updates, specifically dealing with Google Search and the implementation of a new NLP software called BERT. Maybe they changed a little bit of the code on their end during this which could be causing these changes.

[S-Cardenas]

It seems like they're slowly or partially rolling it out since only some of their responses are different.

[pixelicous]

I just started using this library, I never manage to get to download anything. Tried also downloading the chromedriver and specifying it in the attributes.. Nada.. also tried looping till i get downloads, at the third try it always gets hung on "Evaluating", restarting the script seem to get the loop rolling again, till the third try..

Unfortunately all 100 could not be downloaded because some images were not downloadable. 0 is all we got for this search filter!

[S-Cardenas]

@pixelicous Could you check if the HTML being returned from google matches either of formats mentioned in: https://github.com/hardikvasa/google-images-download/issues/280#issuecomment-555182389

[vk379]

This issue just got worse, my aforementioned method of looping until I get the URLs isn't working now, as the URLs are almost never coming up. This is a huge issue. I'm not sure if it's even being worked or not though :(

[galfaroth]

I started to use this lib yesterday and was also unable to download almost anything. Sometimes some keywords worked, but some simple ones like "city" did not succeed.

[pixelicous]

@S-Cardenas Could you point me on how to get the html returned using the library?

From the other replies it seems I am not the only one 😿

[AlNik999]

the same here. Can't load even a single apple image

[S-Cardenas]

@pixelicous I believe the s in https://github.com/hardikvasa/google-images-download/blob/master/google_images_download/google_images_download.py#L714 is the raw html that is returned from the server.

[RiddlerQ]

Same here, if i loop it like 100 times then one image sometimes downloads

[S-Cardenas]

Jakub,

Impressive! What was the issue?

On Wed, Feb 5, 2020 at 4:16 PM Jakub Dobies [email protected] wrote:

After all day of pulling my hairs i finally repaired code. I will upload it after some minor fixes.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hardikvasa/google-images-download/issues/280?email_source=notifications&email_token=ADLS6EPSDNZGWN5CZKN56M3RBMUBDA5CNFSM4JILWRVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK5AJWQ#issuecomment-582616282, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLS6EN33N35QT7EQE7632TRBMUBDANCNFSM4JILWRVA .

[RiddlerQ]

Like you said before "start_line = s.find('rg_meta notranslate')" was the problem.

I deleted my post cause my solution was to simply search for url by finding string that starts with '"https://" and ends with ".jpg"/".png"... , i said that i will upload it but my deadline is tomorrow and my solutions doesn't work with arguments. I created simplifed version with only keywords and number of images to download. It's a lot of work to add all of it, especially that i started learning python 2 days ago ;'c

I will have time after this week so if no one more experienced than me starts to fix it i will try to fully implement parameters etc.

[vk379]

Dude, let me just say that you're a god if you were able to learn python in 2 days and come up with a fix. And honestly, I just need the image URLs man. I am using this library to simply search Images and gifs and get their URLs. Any help is appreciated.

[vk379]

PS: Sorry about accidentally closing and reopening issue!

[RiddlerQ]

I will post simplified version tomorrow and full version in a week or two

[RiddlerQ]

#298 Someone else beat me up to it <3 And its done well not like mine rookie code

[gonjumixproject]

Any update on this one ? Because I have the same issue.. :(

[Igor-Shabalin]

Unfortunately all 10 could not be downloaded because some images were not downloadable. 0 is all we got for this search filter! (((

[miltosc]

same here ...

googleimagesdownload --keywords "house" --limit 120 --chromedriver ./chromedriver -o ./downloaded-images/

result: Unfortunately all 120 could not be downloaded because some images were not downloadable. 0 is all we got for this search filter!

Please help...

Any other info you want please ask for, dont hesitate

[SarthakDandriyal]

Is there a solution for this error. Some other library or some other method to download images from the internet?

[Divyam10]

Use chrome extension guys. All images Download. It can download images over 700 too

[SarthakDandriyal]

@Divyam10 can you guide me how do I use it in python.

[miltosc]

Use chrome extension guys. All images Download. It can download images over 700 too

which extension? send a link

[RiddlerQ]

I thought the problem was solved ;-;

I have my simple downloader on github right now, i wanted to post it earlier but i saw someone here that fixed the issue with google_images_download not finding pics. If someone only needs to download or find urls my code is working every time for me, but its veeeerrryyyy simplified and i applied brute force for searching.

[vk379]

I thought the problem was solved ;-;

I have my simple downloader on github right now, i wanted to post it earlier but i saw someone here that fixed the issue with google_images_download not finding pics. If someone only needs to download or find urls my code is working every time for me, but its veeeerrryyyy simplified and i applied brute force for searching.

Your new library is sick man, I'm already using it. Thanks a bunch dude.

[RiddlerQ]

I thought the problem was solved ;-; I have my simple downloader on github right now, i wanted to post it earlier but i saw someone here that fixed the issue with google_images_download not finding pics. If someone only needs to download or find urls my code is working every time for me, but its veeeerrryyyy simplified and i applied brute force for searching.

Your new library is sick man, I'm already using it. Thanks a bunch dude.

Thanks dude, you totally made my day <3