Caleb Peffer

[Feat] Download files as a part of the request response

3

Customers often want to grab CSV and PDF files that are hosted on a website. Right now, we handle converting PDF files, but what about CSV? A potential solution here...

[BUG] Allow scraping of files (like PDFs) that are more than 10mb

4

We just lost a customer because we weren't able to scrape a pdf on the tesla website. For pdf files, we have a limit of 10mb (right @rafaelsideguide, you mentioned...

[Feat] Improved includeMain content

1

#p3nnywh1stl3 on the discord had a great suggestion for the tags to exclude to get tidy content from a website: ["script", "style", "nav", "header", "footer", ".advertisement", ".sidebar", ".nav", ".menu", "#comments",...

[Feat] Better dev experience when accessing stream results on /crawl

3

Currently, the stream on crawl sends back information in the partial data array, which is limited to 50 items according to the docs. The newer items replace older items in...

[BUG] PDF link in github is failing with a 504 gateway timeout

1

A customer is trying to grab a PDF on GitHub on the /scrape endpoint - they're running into this https://github.com/tgc828/pdftest2/blob/main/dd0f5f631ad54ff8a0a95809a4abed43.pdf Additional error info 504 Gateway Timeout Code: FUNCTION_INVOCATION_TIMEOUT ID: hnd1::qtlxh-1720069704253-7684f544e005

[FEAT] Pages that failed because of Login still charging credits to the account

2

A customer lost 2500 credits because they tried crawling a webpage that had a login - they didn't realize that some of the subpages required the login. I'm not entirely...

A customer brought up the need to bypass authentication on the search endpoint. Suggestion, allow to pass the an auth cookie to the /search endpoint just like they can with...

It seems like running a crawl on https://careerwise.ceric.ca/category/tips-training/ breaks when the extractMainContentOnly parameter is checked. It works otherwise. Notify Josh via crisp once fixed @calebpeffer

Runs the crawler on https://docs.stripe.com/, it discovers around 2800 pages shows the crawl progress, but after it's done crawling, it just returns an array full of null values

[Bug] www.churchofjesuschrist.org/study/manual/missionary-standards-for-disciples-of-jesus-christ?lang=eng doesn't get all links

1

This website only gets a few links with default parameters [www.churchofjesuschrist.org/study/manual/missionary-standards-for-disciples-of-jesus-christ?lang=eng](https://www.churchofjesuschrist.org/study/manual/missionary-standards-for-disciples-of-jesus-christ?lang=eng)