crawl4ai
crawl4ai copied to clipboard
unclecode
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN
Hi, Thanks for this great work. I have been playing around with this, to crawl webpages and get content in markdown format, which can be used to provide to LLMs...
Hey everyone, The final step of development—deployment—is the most challenging. I'm sure many of you will agree with me. Could someone share their experience on the best way to deploy...
Hi, I'm coming back to this issue again. I have my ollama server on the URL as present in the original code. However, `crawl4ai` still uses the local llama that...
In `WebScrappingStrategy`, images are extracted based on the very few possible ways they can get scores in `score_image_for_usefulness`. Having a preferable format name might be one of the easiest, but...
Hi @unclecode - First of all thanks for your amazing work on this lib! It's exciting to have an open source crawler that works quite well! I was testing switching...
I want to scrape a forum to get data for maybe llm model fine-tuning forums have boards boards have threads thread include details like title, author, datetime..etc it's better to...
Is there any way I can pass the input value with the URL, and then Craw4AI performs the search first and then starts crawling data?
Hello and thank you for building this amazing libary. i'm using crawl4ai in a production environment with up to 50 concurrents requests in a fastapi Application. the problem i have...
Are you a developer looking to harness the power of cutting-edge AI models without breaking the bank? Look no further! We're excited to introduce our API service that provides unrestricted...
# Motivation Some deployed environment restricts write permissions. in my case, AWS Lambda restricts write permissions to the `/tmp/` directory only [[Source](https://repost.aws/questions/QUoCCNNS59RryAKeEBezXFcg/how-to-work-around-directory-permissions-in-aws-lambda)].
