feat: add integration of ScrapeGraphAI

[VinciGit00]

This PR will add the integration of ScrapeGraphAI (url)

This PR will:

• reduce the API cost 30% using SGAI instead of Firecrawl
• Increase the speed of response of 25%
• Add proxy rotation and better parallel handling of the HTTP requests

[vercel[bot]]

@VinciGit00 is attempting to deploy a commit to the projects Team on Vercel.

A member of the Team first needs to authorize it.

[banuakkus8]

In my opinion, ScrapeGraph is more suitable than Firecrawl for this purpose

[DPende]

I think the use of ScrapeGraph could bring advantages for this use case

[MarchesiGabriele]

I really think that this would enhance the experience compared to firecrawl.

[vedovati-matteo]

Looking forward to this ScarpegraphAI integration, I think is better suited compared to Firecrawl

[giorgioberardini1]

I was really looking forward to this integration – thank you for making the changes! 🔥 🔥

[markokraemer]

Is ScrapeGraph self-hostable? Because Ideally I want to replace with a in-container scraper

[oldschoola]

Is ScrapeGraph self-hostable? Because Ideally I want to replace with a in-container scraper

looks like it https://github.com/ScrapeGraphAI/Scrapegraph-ai

[Vikrant-Khedkar]

I think this integration will help a lot of people , this is much better than using firecrawl

[VinciGit00]

Is ScrapeGraph self-hostable? Because Ideally I want to replace with a in-container scraper

No for the moment

[scaliseraoul]

I think the use of ScrapeGraph could bring advantages for this use case

[VinciGit00]

Is ScrapeGraph self-hostable? Because Ideally I want to replace with a in-container scraper

looks like it

https://github.com/ScrapeGraphAI/Scrapegraph-ai

It does not scale for high traffic

[f-aguzzi]

ScrapeGraphAI would be a good choice for this use case, being cheaper and faster than FireCrawl

[markokraemer]

I think the use of ScrapeGraph could bring advantages for this use case

I am thinking to include within the sandbox, so 1 agent would have it spun up with its instance as its own scraper

[VinciGit00]

@markokraemer tell me if you need more infos

[markokraemer]

Can u PR it as a seperate tool scrape_graph_search_tool.py or similar, we will introduce modular tools to enable / disable - then we will have it as an option to choose from?

[VinciGit00]

Hi @markokraemer,

I updated it

[markokraemer]

I meant just adding it as a separate tool and keeping the current web-search tool as is.

[markokraemer]

We are adding a Tool Marketplace so we will have in there as well, so you can choose & configure what u want to use

[VinciGit00]

ok where is the tool marketplace?

[VinciGit00]

:/