crawl4ai

crawl4ai copied to clipboard

Dears, while building a tool with your wonderful library, I realized there was no support for disabling SSL verification for websites that either use HTTP only or use HTTPS with...

ifaddict1

Please respect robots.txt

6

when crawling a website the robots.txt should be respected.

Joshix-1

Can you create an option so we can install on pinokio?

1

Hello, can you make a version to easily install it on https://pinokio.computer/?

xenstar

Add Google Vertex AI (i.e. Gemini) in PROVIDER_MODELS of config.py

1

Thank you for the great work and it is prominent! Previously I used Google Vertex AI (i.e. Gemini) for doing something similar to yours but this repository is way better...

huibrian

Related to python Version

1

Please add which python versions are working I am in python 3.8.0 Collecting numpy=1.26.0 (from crawl4ai[torch]) Note: you may need to restart the kernel to use updated packages. ERROR: Could...

y3rawat

Example on whole-blog crawling?

17

Thanks for creating alternatives to [FireCrawl](https://github.com/mendableai/firecrawl) for LLMs! Here is a bit of a question: are there examples or shortcuts for crawling a whole blog (may not may not have...

BradKML

[DOUBT] Performance expectations

3

Does it take advantage of multi threading or something?

Sahil-Gulihar

It would be great if it supports pagination . I wanted to use this solution to scrape whole lengthy documentations of some great open source projects for making easy knowledge...

Greatz08

@unclecode I see that setting hook is not working as expected. I am setting a delay using code below : ``` def delay(driver): print("Delaying for 5 seconds...") time.sleep(5) print("Resuming...") crawler_strategy...

studio-anurag

Using Proxy

5

There's a example code for proxy, but I didn't see the proxy parameter in class AsyncWebCrawler()

shing-li