Search-Engines-Scraper icon indicating copy to clipboard operation
Search-Engines-Scraper copied to clipboard

Published 20 hours ago • verified publisher icon tasos-py

Suggestion: Collect ad based and organic links

Open jatlantic opened this issue 1 year ago • 4 comments

Hi,

Love this package and was wondering if it would be possible to add a feature where one could collect the ad based and organic urls/link results (for Bing for example)?

Cheers, jatlantic

jatlantic avatar Mar 15 '24 22:03 jatlantic

Hi Jatlantic. While ads should be present in HTTP response bodies, the css selectors ere designed to ignore them. So, we should be able to change the selectors and get them, but I don't see the need to add this feature. Of course, you could just create a fork and add it yourself. And if you choose to do so I'll be glad to help

tasos-py avatar Mar 16 '24 16:03 tasos-py

Hi tasos-py, thanks a lot for your swift answer. And yes it makes absolute sense. I'll do a fork to capture organic and ad results for Bing and will go from there.

jatlantic avatar Mar 17 '24 20:03 jatlantic

That's awesome! To help you get started, you'll find SearchEngine objects in "/search_engines/engines/". Each of them has all required selectors in a ._selectors() method that you'd have to edit. Other than that you could maybe add a flag or argument in the parent object to chose between ads and organic results

tasos-py avatar Mar 18 '24 12:03 tasos-py

Good advice, adapted things for a Bing search here: https://github.com/jatlantic/Search-Engines-Scraper. Likely you would do it more elegantly:) Happy to build in your suggestions.

jatlantic avatar Mar 19 '24 11:03 jatlantic