crawl4ai
crawl4ai copied to clipboard
unclecode
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN
Hi UncleCode, I noticed that in `crawl4ai`, the `js_code` execution doesn't seem to happen before the screenshot is taken. It looks like the screenshot is captured without waiting for the...
Does LLMExtractionStrategy support the ability to use AzureChatOpenAI credentials instead of normal open api key?
Currently, the proxy parameter only accepts the server as a proxy setting. However, my proxy requires a username and password for authentication. I would like to request support for adding...
I use the following commands to start: ``` docker pull unclecode/crawl4ai:latest docker run -d -p 8000:80 unclecode/crawl4ai:latest ``` I need to know which configurations and environment variables I can set...
This pull request includes a small but important change to the `crawl4ai/async_crawler_strategy.py` file. The change involves modifying the `crawl` method to allow for an optional wait time before taking a...
I need to take screenshots of websites, but often, pop-ups for privacy settings are in the way. It would be great to detect them and click the button before taking...
Hello, I was wondering if there are plans to support free LLMs maybe with Ollama in Crawl4AI? Currently, the examples and documentation focus on paid APIs like OpenAI, but it...
wait_for=".crypto-row:nth-child(20)", # Wait for 20 rows to load what's format of this param
It would be useful as well to detect pages with: - Captcha - Bot wall - Please update your browser - Parked domain - ... Those errors are already handled...