zimit icon indicating copy to clipboard operation
zimit copied to clipboard
verified publisher icon openzim

Unclear browsertrix exit reason

Open rgaudin opened this issue 11 months ago • 1 comments

In this zimit run, there is no sign of failure in the crawler logs but the crawler exited and zimit reports died with <Signals.SIGKILL: 9> while there is no sign of cancelation.

[zimit::2025-01-16 07:48:37,157] INFO:----------
[zimit::2025-01-16 07:48:37,157] INFO:Testing warc2zim args
[zimit::2025-01-16 07:48:37,157] INFO:Running: warc2zim --name=typemoon.fandom.com_54c6f106 --zim-file=typemoon.fandom.com_54c6f106.zim --publisher=openZIM --scraper-suffix zimit 2.1.7 --output /output --url https://typemoon.fandom.com/wiki/Grand_Servant
[warc2zim::2025-01-16 07:48:37,164] INFO:Arguments valid, no inputs to process. Exiting with return code 100
[zimit::2025-01-16 07:48:37,165] INFO:Writing progress to /output/task_progress.json
[zimit::2025-01-16 07:48:37,172] INFO:
[zimit::2025-01-16 07:48:37,173] INFO:----------
[zimit::2025-01-16 07:48:37,173] INFO:Output to tempdir: /output/.tmpebs92clk - will keep
[zimit::2025-01-16 07:48:37,174] INFO:Running browsertrix-crawler crawl: crawl --failOnFailedSeed --waitUntil load --depth -1 --timeout 90 --behaviors autoplay,autofetch,siteSpecific --behaviorTimeout 90 --sizeLimit 4294967296 --diskUtilization 90 --timeLimit 7200 --url https://typemoon.fandom.com/wiki/Grand_Servant --userAgentSuffix zimit.kiwix.org+ [email protected] --mobileDevice Pixel 2 --cwd /output/.tmpebs92clk --statsFilename /output/crawl.json
{"timestamp":"2025-01-16T07:48:39.873Z","logLevel":"info","context":"general","message":"Browsertrix-Crawler 1.4.2 (with warcio.js 2.4.3)","details":{}}
{"timestamp":"2025-01-16T07:48:39.878Z","logLevel":"info","context":"general","message":"Seeds","details":[{"url":"https://typemoon.fandom.com/wiki/Grand_Servant","scopeType":"prefix","include":["/^https?:\\/\\/typemoon\\.fandom\\.com\\/wiki\\//"],"exclude":[],"allowHash":false,"depth":-1,"sitemap":null,"auth":null,"_authEncoded":null,"maxExtraHops":0,"maxDepth":1000000}]}

...

{"timestamp":"2025-01-16T09:24:00.898Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://typemoon.fandom.com/wiki/Spirit","workerid":0}}
{"timestamp":"2025-01-16T09:24:02.859Z","logLevel":"warn","context":"recorder","message":"Skipping URL from unknown frame","details":{"url":"https://www.fandom.com/silver-surfer.html","frameId":"4382AD716241F05AE101573A95577C0A"}}
{"timestamp":"2025-01-16T09:24:10.539Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":1,"frameUrls":["https://typemoon.fandom.com/wiki/Spirit"],"page":"https://typemoon.fandom.com/wiki/Spirit","workerid":0}}
{"timestamp":"2025-01-16T09:24:10.539Z","logLevel":"info","context":"behavior","message":"Run Script Started","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Spirit","page":"https://typemoon.fandom.com/wiki/Spirit","workerid":0}}
{"timestamp":"2025-01-16T09:24:11.331Z","logLevel":"info","context":"behavior","message":"Run Script Finished","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Spirit","page":"https://typemoon.fandom.com/wiki/Spirit","workerid":0}}
{"timestamp":"2025-01-16T09:24:11.331Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":1,"page":"https://typemoon.fandom.com/wiki/Spirit","workerid":0}}
{"timestamp":"2025-01-16T09:24:13.834Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://typemoon.fandom.com/wiki/Spirit","workerid":0}}
{"timestamp":"2025-01-16T09:24:14.139Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://typemoon.fandom.com/wiki/Ultimate_One"}}
{"timestamp":"2025-01-16T09:24:14.142Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":477,"total":9371,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-01-16T09:24:14.138Z\",\"extraHops\":0,\"url\":\"https:\\/\\/typemoon.fandom.com\\/wiki\\/Ultimate_One\",\"added\":\"2025-01-16T07:49:18.193Z\",\"depth\":2}"]}}
{"timestamp":"2025-01-16T09:24:14.400Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://typemoon.fandom.com/wiki/Ultimate_One","workerid":0}}
{"timestamp":"2025-01-16T09:24:18.310Z","logLevel":"warn","context":"recorder","message":"Skipping URL from unknown frame","details":{"url":"https://www.fandom.com/silver-surfer.html","frameId":"0C889BA63CD9E1BD7E3F4320F2DD0421"}}
{"timestamp":"2025-01-16T09:24:21.646Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":1,"frameUrls":["https://typemoon.fandom.com/wiki/Ultimate_One"],"page":"https://typemoon.fandom.com/wiki/Ultimate_One","workerid":0}}
{"timestamp":"2025-01-16T09:24:21.646Z","logLevel":"info","context":"behavior","message":"Run Script Started","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Ultimate_One","page":"https://typemoon.fandom.com/wiki/Ultimate_One","workerid":0}}
{"timestamp":"2025-01-16T09:24:22.230Z","logLevel":"info","context":"behavior","message":"Run Script Finished","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Ultimate_One","page":"https://typemoon.fandom.com/wiki/Ultimate_One","workerid":0}}
{"timestamp":"2025-01-16T09:24:22.230Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":1,"page":"https://typemoon.fandom.com/wiki/Ultimate_One","workerid":0}}
{"timestamp":"2025-01-16T09:24:24.733Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://typemoon.fandom.com/wiki/Ultimate_One","workerid":0}}
{"timestamp":"2025-01-16T09:24:25.029Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://typemoon.fandom.com/wiki/Tengu"}}
{"timestamp":"2025-01-16T09:24:25.031Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":478,"total":9380,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-01-16T09:24:25.028Z\",\"extraHops\":0,\"url\":\"https:\\/\\/typemoon.fandom.com\\/wiki\\/Tengu\",\"added\":\"2025-01-16T07:49:18.195Z\",\"depth\":2}"]}}
{"timestamp":"2025-01-16T09:24:25.270Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://typemoon.fandom.com/wiki/Tengu","workerid":0}}
{"timestamp":"2025-01-16T09:24:26.844Z","logLevel":"warn","context":"recorder","message":"Skipping URL from unknown frame","details":{"url":"https://www.fandom.com/silver-surfer.html","frameId":"D77E67D1668AF27EA0CDF5AB5B280B00"}}
{"timestamp":"2025-01-16T09:24:31.194Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":1,"frameUrls":["https://typemoon.fandom.com/wiki/Tengu"],"page":"https://typemoon.fandom.com/wiki/Tengu","workerid":0}}
{"timestamp":"2025-01-16T09:24:31.195Z","logLevel":"info","context":"behavior","message":"Run Script Started","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Tengu","page":"https://typemoon.fandom.com/wiki/Tengu","workerid":0}}
{"timestamp":"2025-01-16T09:24:31.964Z","logLevel":"info","context":"behavior","message":"Run Script Finished","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Tengu","page":"https://typemoon.fandom.com/wiki/Tengu","workerid":0}}
{"timestamp":"2025-01-16T09:24:31.965Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":1,"page":"https://typemoon.fandom.com/wiki/Tengu","workerid":0}}
{"timestamp":"2025-01-16T09:24:34.491Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://typemoon.fandom.com/wiki/Tengu","workerid":0}}
{"timestamp":"2025-01-16T09:24:34.781Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://typemoon.fandom.com/wiki/Witch"}}
{"timestamp":"2025-01-16T09:24:34.783Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":479,"total":9389,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-01-16T09:24:34.777Z\",\"extraHops\":0,\"url\":\"https:\\/\\/typemoon.fandom.com\\/wiki\\/Witch\",\"added\":\"2025-01-16T07:49:18.196Z\",\"depth\":2}"]}}
{"timestamp":"2025-01-16T09:24:35.027Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://typemoon.fandom.com/wiki/Witch","workerid":0}}
{"timestamp":"2025-01-16T09:24:36.350Z","logLevel":"warn","context":"recorder","message":"Skipping URL from unknown frame","details":{"url":"https://www.fandom.com/silver-surfer.html","frameId":"B7FEE27B819B92E9A8C73216DC449374"}}
{"timestamp":"2025-01-16T09:24:42.218Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":1,"frameUrls":["https://typemoon.fandom.com/wiki/Witch"],"page":"https://typemoon.fandom.com/wiki/Witch","workerid":0}}
{"timestamp":"2025-01-16T09:24:42.218Z","logLevel":"info","context":"behavior","message":"Run Script Started","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Witch","page":"https://typemoon.fandom.com/wiki/Witch","workerid":0}}
{"timestamp":"2025-01-16T09:24:42.802Z","logLevel":"info","context":"behavior","message":"Run Script Finished","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Witch","page":"https://typemoon.fandom.com/wiki/Witch","workerid":0}}
{"timestamp":"2025-01-16T09:24:42.802Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":1,"page":"https://typemoon.fandom.com/wiki/Witch","workerid":0}}
{"timestamp":"2025-01-16T09:24:45.305Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://typemoon.fandom.com/wiki/Witch","workerid":0}}
{"timestamp":"2025-01-16T09:24:45.737Z","logLevel":"warn","context":"recorder","message":"takeStream interrupted","details":{"size":524285,"url":"https://static.wikia.nocookie.net/typemoon/images/8/82/Alice.png/revision/latest/scale-to-width-down/732?cb=20230624220804","type":"exception","message":"Protocol error (IO.read): Target closed","stack":"TargetCloseError: Protocol error (IO.read): Target closed\n    at CallbackRegistry.clear (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:70:36)\n    at CdpCDPSession._onClosed (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:98:25)\n    at Connection.onMessage (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:127:25)\n    at WebSocket.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/node/NodeWebSocketTransport.js:38:32)\n    at callListener (/app/node_modules/puppeteer-core/node_modules/ws/lib/event-target.js:290:14)\n    at WebSocket.onMessage (/app/node_modules/puppeteer-core/node_modules/ws/lib/event-target.js:209:9)\n    at WebSocket.emit (node:events:518:28)\n    at Receiver.receiverOnMessage (/app/node_modules/puppeteer-core/node_modules/ws/lib/websocket.js:1220:20)\n    at Receiver.emit (node:events:518:28)\n    at Immediate.<anonymous> (/app/node_modules/puppeteer-core/node_modules/ws/lib/receiver.js:601:16)","page":"https://typemoon.fandom.com/wiki/Witch","workerid":0}}
{"timestamp":"2025-01-16T09:24:45.740Z","logLevel":"warn","context":"recorder","message":"Async fetch: possible response size mismatch","details":{"type":"NetworkLoadStreamAsyncFetcher","size":524285,"expected":757580,"url":"https://static.wikia.nocookie.net/typemoon/images/8/82/Alice.png/revision/latest/scale-to-width-down/732?cb=20230624220804","page":"https://typemoon.fandom.com/wiki/Witch","workerid":0}}
{"timestamp":"2025-01-16T09:24:46.502Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://typemoon.fandom.com/wiki/Wizard"}}
{"timestamp":"2025-01-16T09:24:46.504Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":480,"total":9394,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-01-16T09:24:45.582Z\",\"extraHops\":0,\"url\":\"https:\\/\\/typemoon.fandom.com\\/wiki\\/Wizard\",\"added\":\"2025-01-16T07:49:18.197Z\",\"depth\":2}"]}}
{"timestamp":"2025-01-16T09:24:46.749Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://typemoon.fandom.com/wiki/Wizard","workerid":0}}
{"timestamp":"2025-01-16T09:24:46.779Z","logLevel":"warn","context":"recorder","message":"Skipping URL from unknown frame","details":{"url":"https://typemoon.fandom.com/wiki/Wizard","frameId":"A5B62C6ECC4C496BA89BAD0AE2935BDC"}}
{"timestamp":"2025-01-16T09:24:48.658Z","logLevel":"warn","context":"recorder","message":"Skipping URL from unknown frame","details":{"url":"https://www.fandom.com/silver-surfer.html","frameId":"CD60D46DC89952DF267F69ACA64C3793"}}
{"timestamp":"2025-01-16T09:24:54.088Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":1,"frameUrls":["https://typemoon.fandom.com/wiki/Wizard"],"page":"https://typemoon.fandom.com/wiki/Wizard","workerid":0}}
{"timestamp":"2025-01-16T09:24:54.088Z","logLevel":"info","context":"behavior","message":"Run Script Started","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Wizard","page":"https://typemoon.fandom.com/wiki/Wizard","workerid":0}}
{"timestamp":"2025-01-16T09:24:54.751Z","logLevel":"info","context":"behavior","message":"Run Script Finished","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Wizard","page":"https://typemoon.fandom.com/wiki/Wizard","workerid":0}}
{"timestamp":"2025-01-16T09:24:54.752Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":1,"page":"https://typemoon.fandom.com/wiki/Wizard","workerid":0}}
{"timestamp":"2025-01-16T09:24:55.755Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://typemoon.fandom.com/wiki/Wizard","workerid":0}}
{"timestamp":"2025-01-16T09:24:56.041Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://typemoon.fandom.com/wiki/Anti-Hero"}}
{"timestamp":"2025-01-16T09:24:56.046Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":481,"total":9399,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-01-16T09:24:56.037Z\",\"extraHops\":0,\"url\":\"https:\\/\\/typemoon.fandom.com\\/wiki\\/Anti-Hero\",\"added\":\"2025-01-16T07:49:18.200Z\",\"depth\":2}"]}}
{"timestamp":"2025-01-16T09:24:56.325Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://typemoon.fandom.com/wiki/Anti-Hero","workerid":0}}
{"timestamp":"2025-01-16T09:25:03.642Z","logLevel":"warn","context":"recorder","message":"Skipping URL from unknown frame","details":{"url":"https://www.fandom.com/silver-surfer.html","frameId":"F3D0797B9577BF54593652A5E4C6C1FE"}}
{"timestamp":"2025-01-16T09:25:09.046Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":1,"frameUrls":["https://typemoon.fandom.com/wiki/Heroic_Spirit#Anti-Hero"],"page":"https://typemoon.fandom.com/wiki/Anti-Hero","workerid":0}}
{"timestamp":"2025-01-16T09:25:09.046Z","logLevel":"info","context":"behavior","message":"Run Script Started","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Heroic_Spirit#Anti-Hero","page":"https://typemoon.fandom.com/wiki/Anti-Hero","workerid":0}}
{"timestamp":"2025-01-16T09:25:09.914Z","logLevel":"info","context":"behavior","message":"Run Script Finished","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Heroic_Spirit#Anti-Hero","page":"https://typemoon.fandom.com/wiki/Anti-Hero","workerid":0}}
{"timestamp":"2025-01-16T09:25:09.915Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":1,"page":"https://typemoon.fandom.com/wiki/Anti-Hero","workerid":0}}
{"timestamp":"2025-01-16T09:25:12.419Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://typemoon.fandom.com/wiki/Anti-Hero","workerid":0}}
{"timestamp":"2025-01-16T09:25:12.734Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://typemoon.fandom.com/wiki/Yaga"}}
{"timestamp":"2025-01-16T09:25:12.738Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":482,"total":9399,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-01-16T09:25:12.734Z\",\"extraHops\":0,\"url\":\"https:\\/\\/typemoon.fandom.com\\/wiki\\/Yaga\",\"added\":\"2025-01-16T07:49:18.200Z\",\"depth\":2}"]}}
{"timestamp":"2025-01-16T09:25:12.971Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://typemoon.fandom.com/wiki/Yaga","workerid":0}}
{"timestamp":"2025-01-16T09:25:14.643Z","logLevel":"warn","context":"recorder","message":"Skipping URL from unknown frame","details":{"url":"https://www.fandom.com/silver-surfer.html","frameId":"6BD980917537AA1E4005D73BC331E697"}}
{"timestamp":"2025-01-16T09:25:18.938Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":1,"frameUrls":["https://typemoon.fandom.com/wiki/Yaga"],"page":"https://typemoon.fandom.com/wiki/Yaga","workerid":0}}
{"timestamp":"2025-01-16T09:25:18.938Z","logLevel":"info","context":"behavior","message":"Run Script Started","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Yaga","page":"https://typemoon.fandom.com/wiki/Yaga","workerid":0}}
{"timestamp":"2025-01-16T09:25:19.558Z","logLevel":"info","context":"behavior","message":"Run Script Finished","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Yaga","page":"https://typemoon.fandom.com/wiki/Yaga","workerid":0}}
{"timestamp":"2025-01-16T09:25:19.558Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":1,"page":"https://typemoon.fandom.com/wiki/Yaga","workerid":0}}
{"timestamp":"2025-01-16T09:25:22.061Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://typemoon.fandom.com/wiki/Yaga","workerid":0}}
{"timestamp":"2025-01-16T09:25:22.337Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://typemoon.fandom.com/wiki/Catalyst"}}
{"timestamp":"2025-01-16T09:25:22.342Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":483,"total":9411,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-01-16T09:25:22.334Z\",\"extraHops\":0,\"url\":\"https:\\/\\/typemoon.fandom.com\\/wiki\\/Catalyst\",\"added\":\"2025-01-16T07:49:18.202Z\",\"depth\":2}"]}}
{"timestamp":"2025-01-16T09:25:22.589Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://typemoon.fandom.com/wiki/Catalyst","workerid":0}}
{"timestamp":"2025-01-16T09:25:33.604Z","logLevel":"warn","context":"general","message":"Cloudflare check timed out","details":{"seconds":5,"page":"https://typemoon.fandom.com/wiki/Catalyst","workerid":0}}
{"timestamp":"2025-01-16T09:25:51.477Z","logLevel":"error","context":"general","message":"Link extraction timed out","details":{"seconds":5,"page":"https://typemoon.fandom.com/wiki/Catalyst","workerid":0}}
{"timestamp":"2025-01-16T09:25:59.514Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":1,"frameUrls":["https://typemoon.fandom.com/wiki/Servant#Catalyst"],"page":"https://typemoon.fandom.com/wiki/Catalyst","workerid":0}}
{"timestamp":"2025-01-16T09:25:59.515Z","logLevel":"info","context":"behavior","message":"Run Script Started","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Servant#Catalyst","page":"https://typemoon.fandom.com/wiki/Catalyst","workerid":0}}
{"timestamp":"2025-01-16T09:26:02.666Z","logLevel":"warn","context":"recorder","message":"Skipping URL from unknown frame","details":{"url":"https://www.fandom.com/silver-surfer.html","frameId":"31A5F7DAD381FEE33633E2BED224E14C"}}
{"timestamp":"2025-01-16T09:26:04.848Z","logLevel":"info","context":"behavior","message":"Run Script Finished","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Servant#Catalyst","page":"https://typemoon.fandom.com/wiki/Catalyst","workerid":0}}
{"timestamp":"2025-01-16T09:26:04.849Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":1,"page":"https://typemoon.fandom.com/wiki/Catalyst","workerid":0}}
{"timestamp":"2025-01-16T09:26:07.354Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://typemoon.fandom.com/wiki/Catalyst","workerid":0}}
{"timestamp":"2025-01-16T09:26:07.658Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://typemoon.fandom.com/wiki/Command_Spell"}}
{"timestamp":"2025-01-16T09:26:07.660Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":484,"total":9411,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-01-16T09:26:07.657Z\",\"extraHops\":0,\"url\":\"https:\\/\\/typemoon.fandom.com\\/wiki\\/Command_Spell\",\"added\":\"2025-01-16T07:49:18.203Z\",\"depth\":2}"]}}
{"timestamp":"2025-01-16T09:26:07.989Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://typemoon.fandom.com/wiki/Command_Spell","workerid":0}}
{"timestamp":"2025-01-16T09:26:11.204Z","logLevel":"warn","context":"recorder","message":"Skipping URL from unknown frame","details":{"url":"https://www.fandom.com/silver-surfer.html","frameId":"8ADE5117779C9AA36686220B504D2614"}}
{"timestamp":"2025-01-16T09:26:16.944Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":1,"frameUrls":["https://typemoon.fandom.com/wiki/Command_Spell"],"page":"https://typemoon.fandom.com/wiki/Command_Spell","workerid":0}}
{"timestamp":"2025-01-16T09:26:16.945Z","logLevel":"info","context":"behavior","message":"Run Script Started","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Command_Spell","page":"https://typemoon.fandom.com/wiki/Command_Spell","workerid":0}}
{"timestamp":"2025-01-16T09:26:17.731Z","logLevel":"info","context":"behavior","message":"Run Script Finished","details":{"frameUrl":"https://typemoon.fandom.com/wiki/Command_Spell","page":"https://typemoon.fandom.com/wiki/Command_Spell","workerid":0}}
{"timestamp":"2025-01-16T09:26:17.731Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":1,"page":"https://typemoon.fandom.com/wiki/Command_Spell","workerid":0}}
{"timestamp":"2025-01-16T09:26:20.234Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://typemoon.fandom.com/wiki/Command_Spell","workerid":0}}
{"timestamp":"2025-01-16T09:26:21.956Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://typemoon.fandom.com/wiki/Blackening"}}
{"timestamp":"2025-01-16T09:26:21.958Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":485,"total":9549,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-01-16T09:26:20.546Z\",\"extraHops\":0,\"url\":\"https:\\/\\/typemoon.fandom.com\\/wiki\\/Blackening\",\"added\":\"2025-01-16T07:49:18.204Z\",\"depth\":2}"]}}
{"timestamp":"2025-01-16T09:26:22.167Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://typemoon.fandom.com/wiki/Blackening","workerid":0}}
{"timestamp":"2025-01-16T09:26:22.200Z","logLevel":"warn","context":"recorder","message":"Skipping URL from unknown frame","details":{"url":"https://typemoon.fandom.com/wiki/Blackening","frameId":"880C3A1805C0620B0DA12D48D45DAFD8"}}
Traceback (most recent call last):
  File "/usr/bin/zimit", line 8, in <module>
    sys.exit(zimit.zimit())
             ^^^^^^^^^^^^^
  File "/app/zimit/lib/python3.12/site-packages/zimit/zimit.py", line 688, in zimit
    sys.exit(run(sys.argv[1:]))
             ^^^^^^^^^^^^^^^^^
  File "/app/zimit/lib/python3.12/site-packages/zimit/zimit.py", line 574, in run
    raise subprocess.CalledProcessError(crawl.returncode, cmd_args)
subprocess.CalledProcessError: Command '['crawl', '--failOnFailedSeed', '--waitUntil', 'load', '--depth', '-1', '--timeout', '90', '--behaviors', 'autoplay,autofetch,siteSpecific', '--behaviorTimeout', '90', '--sizeLimit', '4294967296', '--diskUtilization', '90', '--timeLimit', '7200', '--url', 'https://typemoon.fandom.com/wiki/Grand_Servant', '--userAgentSuffix', 'zimit.kiwix.org+ [email protected]', '--mobileDevice', 'Pixel 2', '--cwd', '/output/.tmpebs92clk', '--statsFilename', '/output/crawl.json']' died with <Signals.SIGKILL: 9>.
[zimit::2025-01-16 09:26:25,301] INFO:
[zimit::2025-01-16 09:26:25,302] INFO:
[zimit::2025-01-16 09:26:25,302] INFO:SIGINT/SIGTERM received, stopping zimit
[zimit::2025-01-16 09:26:25,302] INFO:
[zimit::2025-01-16 09:26:25,302] INFO:

rgaudin avatar Jan 20 '25 11:01 rgaudin

New occurrences of this:

  • https://farm.zimit.kiwix.org/pipeline/47054b89-a036-4a88-8f23-d99dfc6059ec/debug
  • https://farm.zimit.kiwix.org/pipeline/a06320ae-5b09-48fe-a954-6362a098e5ba/debug

rgaudin avatar Feb 03 '25 13:02 rgaudin