Use Crawlee for adding new docs

[Dreamzlol]

Is your feature request related to a problem? Please describe. Currently, we're facing significant issues when adding documents. Sometimes, it doesn't scrape the entire document, causing us to miss large parts.

Describe the solution you'd like One potential solution could be to use https://crawlee.dev/ I've personally tried it with Svelte, which has some bugs with the Cursor @docs feature, but Crawlee has worked for me, successfully scraping the entire documents even with SSR.

Additional context Feel free to add any other context or screenshots regarding the feature request here.