zimit
zimit copied to clipboard
openzim
Error "Attempted to use detached Frame"
My command:
docker run -v .:/output --name urbandictionary-com-2024-december --rm ghcr.io/openzim/zimit:latest zimit --url "https://urbandictionary.com" --name urbandictionary-com-2024-december --workers 350 --diskUtilization 0
Yes 350 workers - I have a powerful PC with 64 GB of DDR5 RAM, top cpu AMD 7950x
Part of logs:
{"timestamp":"2024-12-20T09:04:24.213Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://www.urbandictionary.com/define.php?term=upsidedown","workerid":239}}
{"timestamp":"2024-12-20T09:04:24.213Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://www.urbandictionary.com/author.php?author=gekyumey","workerid":281}}
{"timestamp":"2024-12-20T09:04:24.213Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://www.urbandictionary.com/author.php?author=lessashamed","workerid":29
6}}
{"timestamp":"2024-12-20T09:04:24.213Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://www.urbandictionary.com/author.php?author=Wollah","workerid":270}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame '0EB3DF7DA8931EB
F5C092A872BF4C16C'.","stack":"Error: Attempted to use detached Frame '0EB3DF7DA8931EBF5C092A872BF4C16C'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n at runNextTicks (node:internal/process/task_queues:60:5)\n at listOnTimeou
t (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\n at async Crawler.crawlPage (file:///app
/dist/crawler.js:586:9)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)"}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"warn","context":"links","message":"Link Extraction failed","details":{"type":"exception","message":"Attempted to use detached Frame '0EB3DF7DA8931EBF5C092A872BF4C16
C'.","stack":"Error: Attempted to use detached Frame '0EB3DF7DA8931EBF5C092A872BF4C16C'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n at file:
///app/dist/crawler.js:1350:26\n at Array.map (<anonymous>)\n at Crawler.extractLinks (file:///app/dist/crawler.js:1348:49)\n at Crawler.loadPage (file:///app/dist/crawler.js:1292:20)\n at runNextTicks (n
ode:internal/process/task_queues:60:5)\n at listOnTimeout (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async Crawler.default [as driver] (file:///app/dist/default
Driver.js:2:5)\n at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)"}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame '0EB3DF7DA8931EBF5C092A872BF4C16C'."
,"stack":"Error: Attempted to use detached Frame '0EB3DF7DA8931EBF5C092A872BF4C16C'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n at runNextTicks (node:internal/process/task_queues:60:5)\n a
t listOnTimeout (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async PageWorker.timedCrawlP
age (file:///app/dist/util/worker.js:173:13)\n at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n at async PageWorker.run (file:///app/dist/util/worker.js:199:13)","page":"https://www.urband
ictionary.com/define.php?term=mr%20brown&page=4","workerid":47}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":2,"page":"https://www.urbandictionary.com/define.php?term=mr%20brown&page=4","workerid
":47}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame '1ECB406F811DCC2
C1FE39432ADCE450A'.","stack":"Error: Attempted to use detached Frame '1ECB406F811DCC2C1FE39432ADCE450A'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n at runNextTicks (node:internal/process/task_queues:60:5)\n at listOnTimeou
t (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\n at async Crawler.crawlPage (file:///app
/dist/crawler.js:586:9)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)"}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"warn","context":"links","message":"Link Extraction failed","details":{"type":"exception","message":"Attempted to use detached Frame '1ECB406F811DCC2C1FE39432ADCE450
A'.","stack":"Error: Attempted to use detached Frame '1ECB406F811DCC2C1FE39432ADCE450A'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n at file:
///app/dist/crawler.js:1350:26\n at Array.map (<anonymous>)\n at Crawler.extractLinks (file:///app/dist/crawler.js:1348:49)\n at Crawler.loadPage (file:///app/dist/crawler.js:1292:20)\n at runNextTicks (n
ode:internal/process/task_queues:60:5)\n at listOnTimeout (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async Crawler.default [as driver] (file:///app/dist/default
Driver.js:2:5)\n at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)"}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame '1ECB406F811DCC2C1FE39432ADCE450A'."
,"stack":"Error: Attempted to use detached Frame '1ECB406F811DCC2C1FE39432ADCE450A'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n at runNextTicks (node:internal/process/task_queues:60:5)\n a
t listOnTimeout (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async PageWorker.timedCrawlP
age (file:///app/dist/util/worker.js:173:13)\n at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n at async PageWorker.run (file:///app/dist/util/worker.js:199:13)","page":"https://www.urband
ictionary.com/define.php?term=mr%20brown&page=2","workerid":155}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":2,"page":"https://www.urbandictionary.com/define.php?term=mr%20brown&page=2","workerid
":155}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame '53B2C9B5AC0A846
8F4CF19756A7F3A97'.","stack":"Error: Attempted to use detached Frame '53B2C9B5AC0A8468F4CF19756A7F3A97'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n at runNextTicks (node:internal/process/task_queues:60:5)\n at listOnTimeou
t (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\n at async Crawler.crawlPage (file:///app
/dist/crawler.js:586:9)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)"}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"warn","context":"links","message":"Link Extraction failed","details":{"type":"exception","message":"Attempted to use detached Frame '53B2C9B5AC0A8468F4CF19756A7F3A9
7'.","stack":"Error: Attempted to use detached Frame '53B2C9B5AC0A8468F4CF19756A7F3A97'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n at file:
///app/dist/crawler.js:1350:26\n at Array.map (<anonymous>)\n at Crawler.extractLinks (file:///app/dist/crawler.js:1348:49)\n at Crawler.loadPage (file:///app/dist/crawler.js:1292:20)\n at runNextTicks (n
ode:internal/process/task_queues:60:5)\n at listOnTimeout (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async Crawler.default [as driver] (file:///app/dist/default
Driver.js:2:5)\n at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)"}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame '53B2C9B5AC0A8468F4CF19756A7F3A97'."
,"stack":"Error: Attempted to use detached Frame '53B2C9B5AC0A8468F4CF19756A7F3A97'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n at runNextTicks (node:internal/process/task_queues:60:5)\n a
t listOnTimeout (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async PageWorker.timedCrawlP
age (file:///app/dist/util/worker.js:173:13)\n at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n at async PageWorker.run (file:///app/dist/util/worker.js:199:13)","page":"https://www.urband
ictionary.com/define.php?term=songwriter%27s","workerid":226}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":2,"page":"https://www.urbandictionary.com/define.php?term=songwriter%27s","workerid":2
26}}
{"timestamp":"2024-12-20T09:04:24.214Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame '0CD38CC9AF9210E
5A31258F6F1956C98'.","stack":"Error: Attempted to use detached Frame '0CD38CC9AF9210E5A31258F6F1956C98'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n at runNextTicks (node:internal/process/task_queues:60:5)\n at listOnTimeou
t (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\n at async Crawler.crawlPage (file:///app
/dist/crawler.js:586:9)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)"}}
{"timestamp":"2024-12-20T09:04:24.216Z","logLevel":"warn","context":"links","message":"Link Extraction failed","details":{"type":"exception","message":"Attempted to use detached Frame '0CD38CC9AF9210E5A31258F6F1956C9
8'.","stack":"Error: Attempted to use detached Frame '0CD38CC9AF9210E5A31258F6F1956C98'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n at file:
///app/dist/crawler.js:1350:26\n at Array.map (<anonymous>)\n at Crawler.extractLinks (file:///app/dist/crawler.js:1348:49)\n at Crawler.loadPage (file:///app/dist/crawler.js:1292:20)\n at runNextTicks (n
ode:internal/process/task_queues:60:5)\n at listOnTimeout (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async Crawler.default [as driver] (file:///app/dist/default
Driver.js:2:5)\n at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)"}}
{"timestamp":"2024-12-20T09:04:24.216Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame '0CD38CC9AF9210E5A31258F6F1956C98'."
,"stack":"Error: Attempted to use detached Frame '0CD38CC9AF9210E5A31258F6F1956C98'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n at runNextTicks (node:internal/process/task_queues:60:5)\n a
t listOnTimeout (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async PageWorker.timedCrawlP
age (file:///app/dist/util/worker.js:173:13)\n at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n at async PageWorker.run (file:///app/dist/util/worker.js:199:13)","page":"https://www.urband
ictionary.com/define.php?term=mr%20brown&page=3","workerid":182}}
{"timestamp":"2024-12-20T09:04:24.216Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":2,"page":"https://www.urbandictionary.com/define.php?term=mr%20brown&page=3","workerid
":182}}
{"timestamp":"2024-12-20T09:04:24.216Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame '79D619504161484
ED626C47808D05CED'.","stack":"Error: Attempted to use detached Frame '79D619504161484ED626C47808D05CED'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n at runNextTicks (node:internal/process/task_queues:60:5)\n at listOnTimeou
t (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\n at async Crawler.crawlPage (file:///app
/dist/crawler.js:586:9)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)"}}
{"timestamp":"2024-12-20T09:04:24.216Z","logLevel":"warn","context":"links","message":"Link Extraction failed","details":{"type":"exception","message":"Attempted to use detached Frame '79D619504161484ED626C47808D05CE
D'.","stack":"Error: Attempted to use detached Frame '79D619504161484ED626C47808D05CED'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n at file:
///app/dist/crawler.js:1350:26\n at Array.map (<anonymous>)\n at Crawler.extractLinks (file:///app/dist/crawler.js:1348:49)\n at Crawler.loadPage (file:///app/dist/crawler.js:1292:20)\n at runNextTicks (n
ode:internal/process/task_queues:60:5)\n at listOnTimeout (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async Crawler.default [as driver] (file:///app/dist/default
Driver.js:2:5)\n at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)"}}
{"timestamp":"2024-12-20T09:04:24.216Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame '79D619504161484ED626C47808D05CED'."
,"stack":"Error: Attempted to use detached Frame '79D619504161484ED626C47808D05CED'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n at runNextTicks (node:internal/process/task_queues:60:5)\n a
t listOnTimeout (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async PageWorker.timedCrawlP
age (file:///app/dist/util/worker.js:173:13)\n at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n at async PageWorker.run (file:///app/dist/util/worker.js:199:13)","page":"https://www.urband
ictionary.com/define.php?term=Mrs%20Brown","workerid":137}}
{"timestamp":"2024-12-20T09:04:24.217Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":2,"page":"https://www.urbandictionary.com/define.php?term=Mrs%20Brown","workerid":137}
}
{"timestamp":"2024-12-20T09:04:24.217Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame '7A94E3CBE62EA1D
AEF3711B8D454102C'.","stack":"Error: Attempted to use detached Frame '7A94E3CBE62EA1DAEF3711B8D454102C'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\
n at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js
:173:13)\n at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n at async Promise.allSettled (index 88)"}}
{"timestamp":"2024-12-20T09:04:24.217Z","logLevel":"warn","context":"links","message":"Link Extraction failed","details":{"type":"exception","message":"Attempted to use detached Frame '7A94E3CBE62EA1DAEF3711B8D454102
C'.","stack":"Error: Attempted to use detached Frame '7A94E3CBE62EA1DAEF3711B8D454102C'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n at file:
///app/dist/crawler.js:1350:26\n at Array.map (<anonymous>)\n at Crawler.extractLinks (file:///app/dist/crawler.js:1348:49)\n at Crawler.loadPage (file:///app/dist/crawler.js:1292:20)\n at async Crawler.d
efault [as driver] (file:///app/dist/defaultDriver.js:2:5)\n at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async
PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)\n at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)"}}
{"timestamp":"2024-12-20T09:04:24.217Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame '7A94E3CBE62EA1DAEF3711B8D454102C'."
,"stack":"Error: Attempted to use detached Frame '7A94E3CBE62EA1DAEF3711B8D454102C'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:
157:21)\n at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)\n at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n at async PageWorker.run (file:///app/dist/util/w
orker.js:199:13)\n at async Promise.allSettled (index 88)\n at async runWorkers (file:///app/dist/util/worker.js:276:5)\n at async Crawler.crawl (file:///app/dist/crawler.js:926:9)","page":"https://www.urban
dictionary.com/define.php?term=mrs+brown","workerid":88}}
{"timestamp":"2024-12-20T09:04:24.217Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":2,"page":"https://www.urbandictionary.com/define.php?term=mrs+brown","workerid":88}}
{"timestamp":"2024-12-20T09:04:24.547Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame '1D92D7BD0F88345
24B3BA723CD369208'.","stack":"Error: Attempted to use detached Frame '1D92D7BD0F8834524B3BA723CD369208'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n at runNextTicks (node:internal/process/task_queues:60:5)\n at listOnTimeou
t (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\n at async Crawler.crawlPage (file:///app
/dist/crawler.js:586:9)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)"}}
{"timestamp":"2024-12-20T09:04:24.547Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame '1D92D7BD0F8834524B3BA723CD369208'."
,"stack":"Error: Attempted to use detached Frame '1D92D7BD0F8834524B3BA723CD369208'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n at runNextTicks (node:internal/process/task_queues:60:5)\n a
t listOnTimeout (node:internal/timers:545:9)\n at process.processTimers (node:internal/timers:519:7)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async PageWorker.timedCrawlP
age (file:///app/dist/util/worker.js:173:13)\n at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n at async PageWorker.run (file:///app/dist/util/worker.js:199:13)","page":"https://www.urband
ictionary.com/remove.form.php?reconsider%5Bdefid_to_remove%5D=14272067","workerid":265}}
{"timestamp":"2024-12-20T09:04:24.548Z","logLevel":"warn","context":"behavior","message":"Waiting for custom page load failed","details":{"type":"exception","message":"Attempted to use detached Frame 'C4D6925CAE2A432
C554DC8AE58B1D361'.","stack":"Error: Attempted to use detached Frame 'C4D6925CAE2A432C554DC8AE58B1D361'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:2
3)\n at Crawler.awaitPageLoad (file:///app/dist/crawler.js:1314:25)\n at Crawler.loadPage (file:///app/dist/crawler.js:1285:20)\n at async Crawler.default [as driver] (file:///app/dist/defaultDriver.js:2:5)\
n at async Crawler.crawlPage (file:///app/dist/crawler.js:586:9)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:157:21)\n at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js
:173:13)\n at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n at async Promise.allSettled (index 191)"}}
{"timestamp":"2024-12-20T09:04:24.548Z","logLevel":"error","context":"worker","message":"Worker Exception","details":{"type":"exception","message":"Attempted to use detached Frame 'C4D6925CAE2A432C554DC8AE58B1D361'."
,"stack":"Error: Attempted to use detached Frame 'C4D6925CAE2A432C554DC8AE58B1D361'.\n at CdpFrame.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/util/decorators.js:94:23)\n at CdpPage.t
itle (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:1084:43)\n at Crawler.crawlPage (file:///app/dist/crawler.js:587:42)\n at async PageWorker.crawlPage (file:///app/dist/util/worker.js:
157:21)\n at async PageWorker.timedCrawlPage (file:///app/dist/util/worker.js:173:13)\n at async PageWorker.runLoop (file:///app/dist/util/worker.js:228:17)\n at async PageWorker.run (file:///app/dist/util/w
orker.js:199:13)\n at async Promise.allSettled (index 191)\n at async runWorkers (file:///app/dist/util/worker.js:276:5)\n at async Crawler.crawl (file:///app/dist/crawler.js:926:9)","page":"https://www.urba
ndictionary.com/remove.form.php?reconsider%5Bdefid_to_remove%5D=13230748","workerid":191}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":171}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":286}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":241}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":17}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":52}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":15}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":211}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":242}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":43}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":183}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":122}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":81}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":25}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":99}}
{"timestamp":"2024-12-20T09:04:24.720Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":314}}
...
...
{"timestamp":"2024-12-20T09:04:24.721Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":182}}
{"timestamp":"2024-12-20T09:04:24.725Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":191}}
{"timestamp":"2024-12-20T09:04:24.725Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":265}}
{"timestamp":"2024-12-20T09:07:50.218Z","logLevel":"info","context":"general","message":"Saving crawl state to: /output/.tmpyirbj5uu/collections/crawl-20241219210812241/crawls/crawl-20241220090424-7dc588181026.yaml",
"details":{}}
{"timestamp":"2024-12-20T09:07:50.598Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":261062,"total":1901133,"pending":22,"failed":140,"limit":{"max":0,"hit":false},"pend
ingPages":["{\"seedId\":1,\"started\":\"2024-12-20T09:04:18.715Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=Parramatta%20Eels\",\"added\":\"2024-12-19T21:56:36.833Z\",\"depth\":
4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:18.594Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=bogey&page=3\",\"added\":\"2024-12-19T21:56:36.617Z\",\"depth\":4}","{\"seed
Id\":1,\"started\":\"2024-12-20T09:04:18.045Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=vegitation\",\"added\":\"2024-12-19T21:56:36.608Z\",\"depth\":4}","{\"seedId\":1,\"start
ed\":\"2024-12-20T09:04:17.769Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=Penny+candy\",\"added\":\"2024-12-19T21:56:35.140Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-1
2-20T09:04:17.993Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/author.php?author=goldendeath1134\",\"added\":\"2024-12-19T21:56:35.667Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09
:04:17.928Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/remove.form.php?reconsider%5Bdefid_to_remove%5D=1087053\",\"added\":\"2024-12-19T21:56:35.141Z\",\"depth\":4}","{\"seedId\":1,\"started\":
\"2024-12-20T09:04:21.190Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=K%20Thanks\",\"added\":\"2024-12-19T21:56:37.044Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T0
9:04:17.750Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=predecessor\",\"added\":\"2024-12-19T21:56:35.139Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:17.994Z\
",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=sup%2C+yo%3F\",\"added\":\"2024-12-19T21:56:35.669Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:19.909Z\",\"extraHop
s\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/author.php?author=MJTS\",\"added\":\"2024-12-19T21:56:36.834Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:21.228Z\",\"extraHops\":0,\"url\":\"ht
tps:\\/\\/www.urbandictionary.com\\/define.php?term=k%20thanks%20bye\",\"added\":\"2024-12-19T21:56:37.044Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:18.028Z\",\"extraHops\":0,\"url\":\"https:\\/\
\/www.urbandictionary.com\\/remove.form.php?reconsider%5Bdefid_to_remove%5D=13607153\",\"added\":\"2024-12-19T21:56:36.607Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:18.028Z\",\"extraHops\":0,\"ur
l\":\"https:\\/\\/www.urbandictionary.com\\/remove.form.php?reconsider%5Bdefid_to_remove%5D=3326420\",\"added\":\"2024-12-19T21:56:36.606Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:21.169Z\",\"ext
raHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/remove.form.php?reconsider%5Bdefid_to_remove%5D=4999156\",\"added\":\"2024-12-19T21:56:37.043Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:
19.732Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/remove.form.php?reconsider%5Bdefid_to_remove%5D=12136914\",\"added\":\"2024-12-19T21:56:36.833Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2
024-12-20T09:04:21.188Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/author.php?author=allvanishing\",\"added\":\"2024-12-19T21:56:37.044Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T
09:04:18.117Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=One%20over\",\"added\":\"2024-12-19T21:56:36.613Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:18.028Z\
",\"extraHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=Vertical\",\"added\":\"2024-12-19T21:56:36.607Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:19.324Z\",\"extraHops\":
0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/define.php?term=scungy\",\"added\":\"2024-12-19T21:56:36.833Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:17.993Z\",\"extraHops\":0,\"url\":\"https:
\\/\\/www.urbandictionary.com\\/author.php?author=Get%20real%2C%20Ya%20All%21%21%21\",\"added\":\"2024-12-19T21:56:35.668Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:18.000Z\",\"extraHops\":0,\"url
\":\"https:\\/\\/www.urbandictionary.com\\/remove.form.php?reconsider%5Bdefid_to_remove%5D=1127164\",\"added\":\"2024-12-19T21:56:35.804Z\",\"depth\":4}","{\"seedId\":1,\"started\":\"2024-12-20T09:04:19.732Z\",\"extr
aHops\":0,\"url\":\"https:\\/\\/www.urbandictionary.com\\/remove.form.php?reconsider%5Bdefid_to_remove%5D=17806886\",\"added\":\"2024-12-19T21:56:36.833Z\",\"depth\":4}"]}}
{"timestamp":"2024-12-20T09:07:50.598Z","logLevel":"info","context":"general","message":"Crawling done","details":{}}
{"timestamp":"2024-12-20T09:07:50.598Z","logLevel":"info","context":"general","message":"Exiting, Crawl status: interrupted","details":{}}
Traceback (most recent call last):
File "/usr/bin/zimit", line 8, in <module>
sys.exit(zimit.zimit())
^^^^^^^^^^^^^
File "/app/zimit/lib/python3.12/site-packages/zimit/zimit.py", line 688, in zimit
run(sys.argv[1:])
File "/app/zimit/lib/python3.12/site-packages/zimit/zimit.py", line 574, in run
raise subprocess.CalledProcessError(crawl.returncode, cmd_args)
subprocess.CalledProcessError: Command '['crawl', '--failOnFailedSeed', '--workers', '350', '--waitUntil', 'load', '--depth', '-1', '--timeout', '90', '--behaviors', 'autoplay,autofetch,siteSpecific', '--behaviorTime
out', '90', '--diskUtilization', '0', '--url', 'https://urbandictionary.com', '--userAgentSuffix', '+Zimit', '--mobileDevice', 'Pixel 2', '--cwd', '/output/.tmpyirbj5uu']' returned non-zero exit status 10.
[zimit::2024-12-20 09:07:50,690] INFO:
[zimit::2024-12-20 09:07:50,690] INFO:----------
[zimit::2024-12-20 09:07:50,690] INFO:Cleanup, removing temp dir: /output/.tmpyirbj5uu
Current version of docker image.
This is beyond usual usage of the crawler, thanks for reporting, always good to know the limits, but I doubt it might be solved soon
Limit you mean 350 workers?
But what this error about? What is detached Frame?