jaredcdep

same issue on a few linux tunnels

same as this which they marked as fixed but the code highlighted in the issue was not changed https://github.com/firecrawl/firecrawl/issues/2056

I am attempting to apply the diffs of https://github.com/firecrawl/firecrawl/pull/2381 onto my local v2.7.0 tag needed to ask co pilot a few things as TS is not my most familiar language

looking now - https://github.com/firecrawl/firecrawl/pull/2364 may be a smaller change - and more maintainable as a temp fork @krim404 by any chance have you tried the change against v2.7.0? Once I...

would `SCRAPEURL_ENGINE_WATERFALL_DELAY_MS` config item affect this issue , or does it only affect calling the main scrape jobs? https://github.com/firecrawl/firecrawl/blob/main/apps/api/src/scraper/scrapeURL/index.ts#L522

I have tested a locally built image, and I think the fix works, but it appears that it just tries to crawl the next url just as fast, I wonder...

had to add extra delay elements to the crawl config - delay does nothing at all ``` { "url": "https://www.abcdef.com/", ... "delay": 2, "maxConcurrency": 1, "scrapeOptions": { "waitFor": 1000, ......

I have added a few quick and dirty changes I need to get a crawl done this weekend: Add a 30s delay when encountering 403 and 429 (there is no...

> thank you for this suggestion. i also ran into rate limiting (but didnt bother so far) and implemented your fix. Maybe you should create a different bug + pullrequest...

@krim404 I have pushed https://github.com/jaredcdep/firecrawl/pull/1 to my fork I had to make a branch from the v2.7.0 tag I also squashed my commits as I had my local gitlab ci...