browsertrix-crawler icon indicating copy to clipboard operation
browsertrix-crawler copied to clipboard
verified publisher icon webrecorder

Crawler exit stuck

Open benoit74 opened this issue 11 months ago • 5 comments

Crawler version : 1.5.1 (will update "soon")

We have a situation where the crawler gets interrupted due to the time limit but never exits. I sent a SIGTERM to the crawler and this is the result (we have multiple occurences of the crawler getting stuck, and for two of them I sent the SIGTERM and got same result as shown below. What is weird is that we've probably been blacklisted because all pages before the time limit seems to be ending with Direct fetch of page URL timed out error.

{"timestamp":"2025-02-11T12:58:32.272Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.cruiserswiki.org/wiki/Special:BookSources/0953926907"}}
{"timestamp":"2025-02-11T12:58:32.274Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":1157,"total":6498,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-02-11T12:58:32.271Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.cruiserswiki.org\\/wiki\\/Special:BookSources\\/0953926907\",\"added\":\"2025-02-11T11:04:42.907Z\",\"depth\":2}"]}}
{"timestamp":"2025-02-11T13:00:02.305Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.cruiserswiki.org/wiki/Special:BookSources/0953926907","workerid":0}}
{"timestamp":"2025-02-11T13:00:02.324Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.cruiserswiki.org/wiki/Talk:Ireland"}}
{"timestamp":"2025-02-11T13:00:02.326Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":1157,"total":6498,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-02-11T13:00:02.323Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.cruiserswiki.org\\/wiki\\/Talk:Ireland\",\"added\":\"2025-02-11T11:04:42.907Z\",\"depth\":2}"]}}
{"timestamp":"2025-02-11T13:01:32.357Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.cruiserswiki.org/wiki/Talk:Ireland","workerid":0}}
{"timestamp":"2025-02-11T13:01:32.380Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.cruiserswiki.org/wiki/Special:WhatLinksHere/Ireland"}}
{"timestamp":"2025-02-11T13:01:32.382Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":1157,"total":6498,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-02-11T13:01:32.379Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.cruiserswiki.org\\/wiki\\/Special:WhatLinksHere\\/Ireland\",\"added\":\"2025-02-11T11:04:42.934Z\",\"depth\":2}"]}}
{"timestamp":"2025-02-11T13:03:02.413Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.cruiserswiki.org/wiki/Special:WhatLinksHere/Ireland","workerid":0}}
{"timestamp":"2025-02-11T13:03:02.418Z","logLevel":"info","context":"general","message":"Time threshold reached 7279.758 > 7200, stopping","details":{}}
{"timestamp":"2025-02-11T13:03:02.430Z","logLevel":"info","context":"general","message":"Crawler interrupted, gracefully finishing current pages","details":{}}
{"timestamp":"2025-02-11T13:03:02.431Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":0}}
{"timestamp":"2025-02-12T20:33:39.133Z","logLevel":"info","context":"general","message":"SIGTERM received...","details":{}}
{"timestamp":"2025-02-12T20:33:39.134Z","logLevel":"info","context":"general","message":"SIGNAL: stopping crawl now...","details":{}}
{"timestamp":"2025-02-12T20:33:39.141Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Limerick","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.141Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Ballycastle","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.141Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Fahan","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Galway","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Sligo","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Westport,_Ireland","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Ballycastle,_N_Ireland","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Bangor","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Belfast_Lough","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Carrickfergus","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Donegal","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Foyle","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Larne","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Portaferry","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Phennick","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Strangford_Lough","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Template:Port","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/File:Transport_map.jpg","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Special:BookSources/0953926907","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Talk:Ireland","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Special:WhatLinksHere/Ireland","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.146Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":0,"frameUrls":[],"page":"https://www.cruiserswiki.org/wiki/Fahan","workerid":0}}
{"timestamp":"2025-02-12T20:33:39.146Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":0,"page":"https://www.cruiserswiki.org/wiki/Fahan","workerid":0}}
{"timestamp":"2025-02-12T20:33:39.150Z","logLevel":"warn","context":"general","message":"Link Extraction failed in frame","details":{"page":"https://www.cruiserswiki.org/wiki/Kilrush","workerid":0,"type":"exception","message":"Protocol error (Runtime.callFunctionOn): Target closed","stack":"TargetCloseError: Protocol error (Runtime.callFunctionOn): Target closed\n    at CallbackRegistry.clear (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:77:36)\n    at CdpCDPSession._onClosed (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:106:25)\n    at #onClose (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:171:21)\n    at Connection.dispose (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:177:22)\n    at CdpBrowser.disconnect (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Browser.js:226:26)\n    at CdpBrowser.close (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Browser.js:222:20)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Browser.close (file:///app/dist/util/browser.js:241:13)\n    at async Crawler.serializeAndExit (file:///app/dist/crawler.js:923:13)\n    at async handleTerminate (file:///app/dist/main.js:30:13)"}}
{"timestamp":"2025-02-12T20:33:39.260Z","logLevel":"info","context":"general","message":"Exiting, Crawl status: interrupted","details":{}}

{"timestamp":"2025-02-12T07:35:08.324Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Strasbourg.htm"}}
{"timestamp":"2025-02-12T07:35:08.325Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":151,"total":1188,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":1,\"started\":\"2025-02-12T07:35:08.323Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.seat61.com\\/international-trains\\/other-cities\\/trains-from-Strasbourg.htm\",\"added\":\"2025-02-12T05:41:02.015Z\",\"depth\":1}"]}}
{"timestamp":"2025-02-12T07:36:38.353Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Strasbourg.htm","workerid":0}}
{"timestamp":"2025-02-12T07:36:38.384Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.seat61.com/international-trains/trains-from-Tallinn.htm"}}
{"timestamp":"2025-02-12T07:36:38.387Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":151,"total":1188,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":1,\"started\":\"2025-02-12T07:36:38.383Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.seat61.com\\/international-trains\\/trains-from-Tallinn.htm\",\"added\":\"2025-02-12T05:41:02.015Z\",\"depth\":1}"]}}
{"timestamp":"2025-02-12T07:38:08.417Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.seat61.com/international-trains/trains-from-Tallinn.htm","workerid":0}}
{"timestamp":"2025-02-12T07:38:08.444Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Toulouse.htm"}}
{"timestamp":"2025-02-12T07:38:08.446Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":151,"total":1188,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":1,\"started\":\"2025-02-12T07:38:08.443Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.seat61.com\\/international-trains\\/other-cities\\/trains-from-Toulouse.htm\",\"added\":\"2025-02-12T05:41:02.017Z\",\"depth\":1}"]}}
{"timestamp":"2025-02-12T07:39:38.452Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Toulouse.htm","workerid":0}}
{"timestamp":"2025-02-12T07:39:38.480Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.seat61.com/international-trains/trains-from-Venice.htm"}}
{"timestamp":"2025-02-12T07:39:38.481Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":151,"total":1188,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":1,\"started\":\"2025-02-12T07:39:38.479Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.seat61.com\\/international-trains\\/trains-from-Venice.htm\",\"added\":\"2025-02-12T05:41:02.018Z\",\"depth\":1}"]}}
{"timestamp":"2025-02-12T07:41:08.509Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.seat61.com/international-trains/trains-from-Venice.htm","workerid":0}}
{"timestamp":"2025-02-12T07:41:08.517Z","logLevel":"info","context":"general","message":"Time threshold reached 7213.174 > 7200, stopping","details":{}}
{"timestamp":"2025-02-12T07:41:08.536Z","logLevel":"info","context":"general","message":"Crawler interrupted, gracefully finishing current pages","details":{}}
{"timestamp":"2025-02-12T07:41:08.536Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":0}}
{"timestamp":"2025-02-12T20:30:50.242Z","logLevel":"info","context":"general","message":"SIGTERM received...","details":{}}
{"timestamp":"2025-02-12T20:30:50.243Z","logLevel":"info","context":"general","message":"SIGNAL: stopping crawl now...","details":{}}
{"timestamp":"2025-02-12T20:30:50.263Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/european-train-tickets-by-phone.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/european-train-booking-horizons.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/european-travel-from-uk-regions.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/interrail-and-eurail-reservations.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/interrail-and-eurail-passes-on-eurostar.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/ferry-to-spain.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Algarve.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Madrid.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Amsterdam.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Athens.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Barcelona.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Switzerland.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Belgrade.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Berlin.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Bordeaux.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Bruges.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Bratislava.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Brussels.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Bucharest.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Budapest.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Nice.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Cesky-Krumlov.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Cinque-Terre.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Cologne.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Copenhagen.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Dublin.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Dubrovnik.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Florence.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Frankfurt.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Gibraltar.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Hallstatt.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Hamburg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Helsinki.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Innsbruck.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Istanbul.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Kyiv.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Krakow.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Lisbon.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Ljubljana.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Luxembourg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Kiev.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Lyon.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Marseille.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Milan.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Montpellier.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Moscow.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Munich.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Naples.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Oslo.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Paris.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Passau.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Pisa.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Porto.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Prague.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Riga.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Rome.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-St-Petersburg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Salzburg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-San-Sebastian.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Sofia.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.272Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Stockholm.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.272Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Strasbourg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.272Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Tallinn.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.273Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Toulouse.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.273Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Venice.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.283Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":0,"frameUrls":[],"page":"https://www.seat61.com/european-train-booking-horizons.htm","workerid":0}}
{"timestamp":"2025-02-12T20:30:50.320Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":0,"page":"https://www.seat61.com/european-train-booking-horizons.htm","workerid":0}}
{"timestamp":"2025-02-12T20:30:50.566Z","logLevel":"info","context":"general","message":"Exiting, Crawl status: interrupted","details":{}}

I have another stuck task which is a bit different:

{"timestamp":"2025-02-12T08:56:22.845Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://bananaguide.com/goToSite/2800","workerid":0}}
{"timestamp":"2025-02-12T08:56:23.573Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://bananaguide.com/article/67290/you-love-jack-review"}}
{"timestamp":"2025-02-12T08:56:23.575Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":917,"total":4682,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-02-12T08:56:22.877Z\",\"extraHops\":0,\"url\":\"https:\\/\\/bananaguide.com\\/article\\/67290\\/you-love-jack-review\",\"added\":\"2025-02-12T07:48:33.215Z\",\"depth\":3}"]}}
{"timestamp":"2025-02-12T08:56:24.963Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:25.299Z","logLevel":"warn","context":"recorder","message":"Skipping URL from unknown frame","details":{"url":"https://bananaguide.com/article/67290/you-love-jack-review","frameId":"77D42A85B5CCDF255253DE5BCC2CE7FA"}}
{"timestamp":"2025-02-12T08:56:27.136Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":1,"frameUrls":["https://bananaguide.com/article/67290/you-love-jack-review"],"page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:27.136Z","logLevel":"info","context":"behavior","message":"Run Script Started","details":{"frameUrl":"https://bananaguide.com/article/67290/you-love-jack-review","page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:27.658Z","logLevel":"info","context":"behavior","message":"Run Script Finished","details":{"frameUrl":"https://bananaguide.com/article/67290/you-love-jack-review","page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:27.659Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":1,"page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:28.660Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:28.664Z","logLevel":"info","context":"general","message":"Size threshold reached 4307928621 >= 4294967296, stopping","details":{}}
{"timestamp":"2025-02-12T08:56:28.692Z","logLevel":"info","context":"general","message":"Crawler interrupted, gracefully finishing current pages","details":{}}
{"timestamp":"2025-02-12T08:56:28.692Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":0}}
{"timestamp":"2025-02-12T08:57:24.804Z","logLevel":"warn","context":"recorder","message":"Failed to load response body","details":{"url":"https://www.lucasentertainment.com/tour/?nats=MTAwMDA0MC4yNy4yLjU1LjAuMC4wLjAuMA","networkId":"00904D649DE4D581E0F7CD7BF89BAAA8","type":"exception","message":"Protocol error (Fetch.getResponseBody): Target closed","stack":"TargetCloseError: Protocol error (Fetch.getResponseBody): Target closed\n    at CallbackRegistry.clear (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:77:36)\n    at CdpCDPSession._onClosed (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:106:25)\n    at #onClose (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:171:21)\n    at WebSocket.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/node/NodeWebSocketTransport.js:43:30)\n    at callListener (/app/node_modules/puppeteer-core/node_modules/ws/lib/event-target.js:290:14)\n    at WebSocket.onClose (/app/node_modules/puppeteer-core/node_modules/ws/lib/event-target.js:220:9)\n    at WebSocket.emit (node:events:518:28)\n    at WebSocket.emitClose (/app/node_modules/puppeteer-core/node_modules/ws/lib/websocket.js:272:10)\n    at Socket.socketOnClose (/app/node_modules/puppeteer-core/node_modules/ws/lib/websocket.js:1341:15)\n    at Socket.emit (node:events:518:28)","page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T20:34:03.730Z","logLevel":"info","context":"general","message":"SIGTERM received...","details":{}}
{"timestamp":"2025-02-12T20:34:03.732Z","logLevel":"info","context":"general","message":"SIGNAL: stopping crawl now...","details":{}}
{"timestamp":"2025-02-12T20:34:03.735Z","logLevel":"info","context":"general","message":"Exiting, Crawl status: interrupted","details":{}}

benoit74 avatar Feb 12 '25 20:02 benoit74

Issue still present on 1.5.4 ; I strongly suspect this is somehow linked to new retry logic and the usage of sizeLimit / timeLimit. How can I help to further diagnose the problem?

benoit74 avatar Feb 24 '25 18:02 benoit74

Hm, just to confirm, the crawler prints Exiting, Crawl status: interrupted on its own but does not exit from there?

ikreymer avatar Feb 26 '25 20:02 ikreymer

Have not seen that before - and I don't think its related to retries, since all that happens after is:

    logger.info(`Exiting, Crawl status: ${status}`);

    await this.closeLog();

    if (this.crawlState && status) {
      await this.crawlState.setStatus(status);
    }
    process.exit(exitCode);

We could add a timeout to closeLog() and setStatus(), have not seen any issues stalling there before..

ikreymer avatar Feb 26 '25 20:02 ikreymer

Hm, just to confirm, the crawler prints Exiting, Crawl status: interrupted on its own but does not exit from there?

No, sorry, look at timestamps, it got stuck at Failed to load response body. Then I sent a SIGTERM and it printed the last two lines. I kept them as reference in the logs to indicate that process was really still alive, waiting for something to happen

benoit74 avatar Feb 26 '25 21:02 benoit74

And first task got stuck at Worker done, all tasks complete

benoit74 avatar Feb 26 '25 21:02 benoit74