firecrawl
[BUG] worker-1 exited with code 1 and api-1 exited with code 1
Describe the Bug
I tried to start Firecrawl from docker.
$ docker compose up
WARN[0000] docker-compose.yaml: `version` is obsolete
[+] Running 5/5
✔ Network firecrawl_backend Created 0.1s
✔ Container firecrawl-redis-1 Created 0.0s
✔ Container firecrawl-playwright-service-1 Created 0.0s
✔ Container firecrawl-api-1 Created 0.0s
✔ Container firecrawl-worker-1 Created 0.0s
Attaching to api-1, playwright-service-1, redis-1, worker-1
redis-1 | 1:C 21 Jul 2024 02:24:56.298 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis-1 | 1:C 21 Jul 2024 02:24:56.298 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
redis-1 | 1:C 21 Jul 2024 02:24:56.298 * Configuration loaded
redis-1 | 1:M 21 Jul 2024 02:24:56.299 * monotonic clock: POSIX clock_gettime
redis-1 | 1:M 21 Jul 2024 02:24:56.299 * Running mode=standalone, port=6379.
redis-1 | 1:M 21 Jul 2024 02:24:56.300 * Server initialized
redis-1 | 1:M 21 Jul 2024 02:24:56.300 * Ready to accept connections tcp
playwright-service-1 | [2024-07-21 02:24:58 +0000] [9] [INFO] Running on http://[::]:3000 (CTRL + C to quit)
worker-1 | /usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:22762
worker-1 | throw new Error(
worker-1 | ^
worker-1 |
worker-1 | Error: Error when performing the request to https://registry.npmjs.org/pnpm/latest; for troubleshooting help, see https://github.com/nodejs/corepack#troubleshooting
worker-1 | at fetch (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:22762:11)
worker-1 | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
worker-1 | at async fetchAsJson (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:22776:20)
worker-1 | ... 4 lines matching cause stack trace ...
worker-1 | at async Object.runMain (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:24235:5) {
worker-1 | [cause]: TypeError: fetch failed
worker-1 | at node:internal/deps/undici/undici:12502:13
worker-1 | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
worker-1 | at async fetch (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:22756:16)
worker-1 | at async fetchAsJson (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:22776:20)
worker-1 | at async fetchLatestStableVersion (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:22703:20)
worker-1 | at async fetchLatestStableVersion2 (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:22826:14)
worker-1 | at async Engine.getDefaultVersion (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:23436:23)
worker-1 | at async Engine.executePackageManagerRequest (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:23528:47)
worker-1 | at async Object.runMain (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:24235:5) {
worker-1 | [cause]: ConnectTimeoutError: Connect Timeout Error (attempted addresses: 104.16.25.34:443, 104.16.29.34:443)
worker-1 | at onConnectTimeout (node:internal/deps/undici/undici:6635:28)
worker-1 | at node:internal/deps/undici/undici:6587:50
worker-1 | at Immediate._onImmediate (node:internal/deps/undici/undici:6619:13)
worker-1 | at process.processImmediate (node:internal/timers:478:21) {
worker-1 | code: 'UND_ERR_CONNECT_TIMEOUT'
worker-1 | }
worker-1 | }
worker-1 | }
worker-1 |
worker-1 | Node.js v20.15.1
worker-1 exited with code 1
api-1 | /usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:22762
api-1 | throw new Error(
api-1 | ^
api-1 |
api-1 | Error: Error when performing the request to https://registry.npmjs.org/pnpm/latest; for troubleshooting help, see https://github.com/nodejs/corepack#troubleshooting
api-1 | at fetch (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:22762:11)
api-1 | at async fetchAsJson (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:22776:20)
api-1 | ... 4 lines matching cause stack trace ...
api-1 | at async Object.runMain (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:24235:5) {
api-1 | [cause]: TypeError: fetch failed
api-1 | at node:internal/deps/undici/undici:12502:13
api-1 | at async fetch (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:22756:16)
api-1 | at async fetchAsJson (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:22776:20)
api-1 | at async fetchLatestStableVersion (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:22703:20)
api-1 | at async fetchLatestStableVersion2 (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:22826:14)
api-1 | at async Engine.getDefaultVersion (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:23436:23)
api-1 | at async Engine.executePackageManagerRequest (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:23528:47)
api-1 | at async Object.runMain (/usr/local/lib/node_modules/corepack/dist/lib/corepack.cjs:24235:5) {
api-1 | [cause]: HeadersTimeoutError: Headers Timeout Error
api-1 | at Timeout.onParserTimeout [as callback] (node:internal/deps/undici/undici:7569:32)
api-1 | at Timeout.onTimeout [as _onTimeout] (node:internal/deps/undici/undici:6659:17)
api-1 | at listOnTimeout (node:internal/timers:573:17)
api-1 | at process.processTimers (node:internal/timers:514:7) {
api-1 | code: 'UND_ERR_HEADERS_TIMEOUT'
api-1 | }
api-1 | }
api-1 | }
api-1 |
api-1 | Node.js v20.15.1
api-1 exited with code 1
Worker-1 and api-1 didn't work. I can build it， but I can't run it.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
af0042698c05 firecrawl-playwright-service "/bin/sh -c 'hyperco…" 27 minutes ago Up 27 minutes firecrawl-playwright-service-1
a9ea3b5d74c8 redis:alpine "docker-entrypoint.s…" 27 minutes ago Up 27 minutes 6379/tcp firecrawl-redis-1
$ curl https://registry.npmjs.org/pnpm/latest
{
"name": "pnpm",
"version": "9.5.0",
"keywords": ["pnpm9", "dependency manager", "install", "installer", "uninstall", "remove", "link", "prune", "shrinkwrap", "lockfile", "fast", "rapid", "efficient", "package.json", "packages", "dependencies", "symlinks", "hardlinks", "modules", "npm", "package manager", "monorepo", "multi-package", "workspace:*"],
"license": "MIT",
"_id": "[email protected]",
"maintainers": [{
"name": "zkochan",
"email": "[email protected]"
}, {
"name": "pnpmuser",
"email": "[email protected]"
}],
"homepage": "https://pnpm.io",
"bugs": {
"url": "https://github.com/pnpm/pnpm/issues"
},
"bin": {
"pnpm": "bin/pnpm.cjs",
"pnpx": "bin/pnpx.cjs"
},
"dist": {
"shasum": "8c155dc114e1689d18937974f6571e0ceee66f1d",
"tarball": "https://registry.npmjs.org/pnpm/-/pnpm-9.5.0.tgz",
"fileCount": 880,
"integrity": "sha512-FAA2gwEkYY1iSiGHtQ0EKJ1aCH8ybJ7fwMzXM9dsT1LDoxPU/BSHlKKp2BVTAWAE5nQujPhQZwJopzh/wiDJAw==",
"signatures": [{
"sig": "MEQCICzcf5DogqT1evYK9hX/oKA1fg0scwuP+oCzvb1rd26kAiB2bRnpayysjOt5HLoX3oVpCX+8vFJ0eDA3qQBpHIrDCA==",
"keyid": "SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"
}],
"unpackedSize": 17664848
},
"main": "bin/pnpm.cjs",
"_from": "file:pnpm-9.5.0.tgz",
"unpkg": "dist/pnpm.cjs",
"engines": {
"node": ">=18.12"
},
"exports": {
".": "./package.json"
},
"funding": "https://opencollective.com/pnpm",
"scripts": {
"lint": "eslint \"src/**/*.ts\" \"test/**/*.ts\"",
"test": "pnpm run compile && pnpm run _test",
"_test": "cross-env PNPM_REGISTRY_MOCK_PORT=7776 jest",
"start": "tsc --watch",
"bundle": "ts-node bundle.ts",
"compile": "tsc --build && pnpm run lint --fix && rimraf dist bin/nodes && pnpm run bundle && shx cp -r node-gyp-bin dist/node-gyp-bin && shx cp -r node_modules/@pnpm/tabtab/lib/templates dist/templates && shx cp -r node_modules/ps-list/vendor dist/vendor && shx cp pnpmrc dist/pnpmrc",
"_compile": "tsc --build",
"pretest:e2e": "rimraf node_modules/.bin/pnpm"
},
"_npmUser": {
"name": "pnpmuser",
"email": "[email protected]"
},
"_resolved": "/tmp/6105e9d949f50a3bb187ab3ae86654d6/pnpm-9.5.0.tgz",
"_integrity": "sha512-FAA2gwEkYY1iSiGHtQ0EKJ1aCH8ybJ7fwMzXM9dsT1LDoxPU/BSHlKKp2BVTAWAE5nQujPhQZwJopzh/wiDJAw==",
"repository": {
"url": "git+https://github.com/pnpm/pnpm.git",
"type": "git"
},
"_npmVersion": "10.7.0",
"description": "Fast, disk space efficient package manager",
"directories": {
"test": "test"
},
"_nodeVersion": "18.20.3",
"preferGlobal": true,
"publishConfig": {
"tag": "next-9",
"executableFiles": ["./dist/node-gyp-bin/node-gyp", "./dist/node-gyp-bin/node-gyp.cmd", "./dist/node_modules/node-gyp/bin/node-gyp.js"]
},
"_hasShrinkwrap": false,
"_npmOperationalInternal": {
"tmp": "tmp/pnpm_9.5.0_1720370654593_0.5246475969538305",
"host": "s3://npm-registry-packages"
}
}
https://registry.npmjs.org/pnpm/latest is accessible.
So, what causes this phenomenon？
Hey @Gedanke ! I just reviewed our docker implementation and wasn't able to reproduce this error; I was able to run both scrape and crawl with no problems. This might not be relevant, but which system are you using? Could you please try updating your Docker and rebuild the containers without cache?
@Gedanke Are you using a proxy? I am also using a proxy and have the same issue.
@Gedanke Are you using a proxy? I am also using a proxy and have the same issue.
Hey @Gedanke ! I just reviewed our docker implementation and wasn't able to reproduce this error; I was able to run both scrape and crawl with no problems. This might not be relevant, but which system are you using? Could you please try updating your Docker and rebuild the containers without cache?
Thank you very much! This case is running on my computer wsl2 system. Yesterday I used source code to build firecrawl, which can provide normal service. But these days, I can't launch it from docker. My computer has been using the agent, and I was still able to build docker normally around the beginning of July.
@Gedanke Are you using a proxy? I am also using a proxy and have the same issue.
Sure, my computer has been using the agent, about the beginning of July, I can still build docker normally. But recently I couldn't launch it from docker. The same problem can occur on another server that does not use a proxy.
@dgedanke @cypher256 are you still facing this issue? Added stale tag as we didn't receive any comment on this issue for more than 40 days.
@dgedanke @cypher256 are you still facing this issue? Added stale tag as we didn't receive any comment on this issue for more than 40 days.
I'm sorry for taking so long to reply. I may have found the problem last month. My equipment environment does not visit https://registry.npmjs.org. To do this, I set up a mirror to NPM https://registry.npmmirror.com.
RUN corepack enable this order will make pnpm access the official https://registry.npmmirror.com, not my default https://registry.npmjs.org.
Even if I add RUN NPM config set registry https://registry.npmmirror.com at the end, or to write https://registry.npmmirror.com into the configuration file, the new image is still will not take effect.
I ended up solving the problem that way.
# RUN corepack enable
# use npm to install pnpm, skip corepack
RUN npm config set registry https://registry.npmmirror.com
RUN npm install -g pnpm && \
pnpm config set registry https://registry.npmmirror.com
And,
$ curl http://127.0.0.1:3002/test
Hello, world!
But it's not a good choice. Shutting down corepack and losing npm package management could be potentially risky.