🐛 BUG: pages publish very slow / crashing

[revmischa]

What version of Wrangler are you using?

2.0.8

What operating system are you using?

Mac

Describe the Bug

I'm trying to upload my site, it's about 300MB, it uploads to KV fine with wrangler 1

When I upload to pages with wrangler 2 it goes reaaallly slow and then crashes

▲ [WARNING] Warning: Your working directory is a git repo and has uncommitted changes

  To silence this warning, pass in --commit-dirty=true


🌍  Uploading... (809/8009)

/usr/local/lib/node_modules/wrangler/wrangler-dist/cli.js:16264
            throw ex;
            ^

FatalError: Failed to upload files. Please try again.
    at /usr/local/lib/node_modules/wrangler/wrangler-dist/cli.js:123689:31
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async run2 (/usr/local/lib/node_modules/wrangler/wrangler-dist/cli.js:122127:26) {
  code: 1
}

I posted it on discord https://discord.com/channels/595317990191398933/799437470004412476/983442112534175815

If I try to upload it via web it errors telling me not to upload more than 1000 files. If I zip it it says the zip is too big. I don't see a way to upload a zip file with the CLI. No idea how to get my files into Pages.

[sidharthachatterjee]

Hey there! Sorry to hear this. What you're describing with the dashboard is expected (we'll be bumping that up soon).

What you're seeing with wrangler however is a bug. Our hypothesis is that your auth token is expiring during the upload because of low upload bandwidth? I've opened a PR with a fix for this.

Can you try this version of wrangler and let us know if it helps?

npx https://prerelease-registry.developers.workers.dev/runs/2455435541/npm-package-wrangler-1199 dev path/to/script.js

Cheers.

[ElijahKotyluk]

@sidharthachatterjee I have tested this a couple times, I am having no issues when running the dev command you provided, but still am not seeing any improvements when trying to run the pages publish command. It still hangs and then throws the same error as above.

[revmischa]

I ran npx https://prerelease-registry.developers.workers.dev/runs/2455435541/npm-package-wrangler-1199 pages publish public and had the same issue

[sidharthachatterjee]

Interesting. Thank you for testing this. In that case, my hypothesis was incorrect. I'll ship that anyway since it's an improvement.

I'll add some verbose debug logs and ship a version for you'll so we can get some more info on what's up. Cheers.

[ElijahKotyluk]

Thank you for actively working on this.

[sidharthachatterjee]

Hi folks! Can you please try npx https://prerelease-registry.developers.workers.dev/runs/2469110657/npm-package-wrangler-1217 pages publish <dir> and post your logs here?

This will print some debugging information for us (HTTP requests, how long they took, the response code)

[ElijahKotyluk]

@sidharthachatterjee fresh logs:

▲ [WARNING] HTTP 200 /accounts/:account_id/pages/projects/:project/upload-token 0.50s


▲ [WARNING] HTTP 200 /pages/assets/check-missing 1.36s


▲ [WARNING] HTTP 200 /pages/assets/upload 139.55s


▲ [WARNING] HTTP 200 /pages/assets/upload 153.21s


▲ [WARNING] HTTP 200 /pages/assets/upload 172.97s


▲ [WARNING] HTTP 200 /pages/assets/upload 145.21s


▲ [WARNING] HTTP 200 /pages/assets/upload 138.10s


▲ [WARNING] HTTP 200 /pages/assets/upload 128.20s


▲ [WARNING] HTTP 403 /pages/assets/upload 129.70s


▲ [WARNING] HTTP 403 /pages/assets/upload 153.34s


▲ [WARNING] HTTP 403 /pages/assets/upload 144.38s


▲ [WARNING] HTTP 403 /pages/assets/upload 128.99s


▲ [WARNING] HTTP 403 /pages/assets/upload 127.93s

▲ [WARNING] HTTP 403 /pages/assets/upload 144.57s


▲ [WARNING] HTTP 403 /pages/assets/upload 134.71s


▲ [WARNING] HTTP 403 /pages/assets/upload 154.54s


▲ [WARNING] HTTP 403 /pages/assets/upload 142.15s


▲ [WARNING] HTTP 403 /pages/assets/upload 148.50s


▲ [WARNING] HTTP 403 /pages/assets/upload 152.17s


▲ [WARNING] HTTP 403 /pages/assets/upload 136.69s


▲ [WARNING] HTTP 403 /pages/assets/upload 138.48s


🌍  Uploading... (255/7962)

/Users/ek/.npm/_npx/8be3397ebf390916/node_modules/wrangler/wrangler-dist/cli.js:16264
            throw ex;
            ^

FatalError: Failed to upload files. Please try again.
    at /Users/ek/.npm/_npx/8be3397ebf390916/node_modules/wrangler/wrangler-dist/cli.js:123709:31
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async run2 (/Users/elijahkotyluk/.npm/_npx/8be3397ebf390916/node_modules/wrangler/wrangler-dist/cli.js:122141:26) {
  code: 1
}

[sidharthachatterjee]

Okay, interesting. So that would imply that the JWT expiry fix didn't work. Can you try this now?

npx https://prerelease-registry.developers.workers.dev/runs/2469819200/npm-package-wrangler-1217

Added a few more logs.

[revmischa]

Using npx https://prerelease-registry.developers.workers.dev/runs/2469819200/npm-package-wrangler-1217 pages publish public

Result:

▲ [WARNING] HTTP 403 /pages/assets/upload 1.09s


▲ [WARNING] Expired JWT


▲ [WARNING] HTTP 403 /pages/assets/upload 1.40s


▲ [WARNING] Expired JWT


🌎  Uploading... (809/8009)

/Users/cyber/.npm/_npx/727f611967a5fdf5/node_modules/wrangler/wrangler-dist/cli.js:16264
            throw ex;
            ^

FatalError: Failed to upload files. Please try again.
    at /Users/cyber/.npm/_npx/727f611967a5fdf5/node_modules/wrangler/wrangler-dist/cli.js:123705:31
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async run2 (/Users/cyber/.npm/_npx/727f611967a5fdf5/node_modules/wrangler/wrangler-dist/cli.js:122136:26) {
  code: 1

It really seems to be much much slower than the K/V upload btw

[revmischa]

I'm able to upload everything now eventually but I think it's only because I upgraded my internet connection. It still seems to take a bit longer than it ought to

My site is 339MB I'm uploading at an average 2.5MB/s Should take ~135s

But I get: ✨ Success! Uploaded 8009 files (215.32 sec)

[ElijahKotyluk]

npx https://prerelease-registry.developers.workers.dev/runs/2469819200/npm-package-wrangler-1217 pages publish is giving me:

▲ [WARNING] HTTP 403 /pages/assets/upload 14.28s


▲ [WARNING] Expired JWT


▲ [WARNING] HTTP 403 /pages/assets/upload 15.91s


▲ [WARNING] Expired JWT


🌏  Uploading... (167/7960)

/Users/ek/.npm/_npx/8be3397ebf390916/node_modules/wrangler/wrangler-dist/cli.js:16264
            throw ex;
            ^

FatalError: Failed to upload files. Please try again.
    at /Users/ek/.npm/_npx/8be3397ebf390916/node_modules/wrangler/wrangler-dist/cli.js:123709:31
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async run2 (/Users/ek/.npm/_npx/8be3397ebf390916/node_modules/wrangler/wrangler-dist/cli.js:122141:26) {
  code: 1
}

[ElijahKotyluk]

@sidharthachatterjee I took some time to debug and find a solve for the expiring JWT on a fork, this seems to only be an issue for my larger projects I am deploying through direct uploads. Even with the JWT issue resolved, the uploads are still taking quite a bit longer than I would hope.

[GregBrimble]

Hi @ElijahKotyluk , glad to hear you got it working on a fork. If you can, please put up a PR with your changes, but we have added some refresh logic a little while ago, so maybe this won't be necessary anymore with the latest wrangler versions?

We'll close this issue out in the meantime, but please reopen if there's an ongoing issue. Thanks!

[ElijahKotyluk]

Hi @GregBrimble, thank you for the response. I just tested the latest and still hit issues with the refetch and speed.

I will get something up today, need to spend some time figuring out the test I broke.

The issue with the current implementation is that the way the error is being checked in the case the JWT expires won't pass, the error caught is a ParseError and the error.code that's being checked does not exist on the error caught, which results in the JWT not getting refetched. I just wrote a helper function isJWTExpired: (error: unknown) => boolean but that broke the refetch JWT test.

[GregBrimble]

Ah, okay, thanks for that additional info! Absolutely feel free to put up whatever you have right now and we can help get it over the line 😊

[ElijahKotyluk]

@GregBrimble, PR is up. Thanks again.

[maisfeldchen]

Hi, i'm having the exact same problem. Any chance for a fix in the near future?

[rohit-gohri]

I'm having the same issue with a 90MB project, with around 5840 files

[aexol]

same here

[ElijahKotyluk]

Still seeing this happen on a regular basis, starting to become a bit of a slowdown for development.

[adamsiwiec]

Can not upload a ~300 file project. Maybe the CLI should gzip a project then unzip it on the server?

[mbouclas]

Having the same issue. Publishes are hit or miss. Sometimes they finish, most times they don't. Here's what a failed attempt looks like

[PBhadoo]

This puts the entire project on hold. Tried Several times.

🌎  Uploading... (380/4308)

C:\Users\bhadoo\AppData\Roaming\npm\node_modules\wrangler\wrangler-dist\cli.js:30632
            throw a;
            ^

FatalError: Failed to upload files. Please try again. Error: {"text":"Received a malformed response from the API","notes":[{"text":"Expired JWT"},{"text":"POST /pages/assets/upload -> 403 Forbidden"}],"kind":"error","name":"ParseError"})
    at C:\Users\bhadoo\AppData\Roaming\npm\node_modules\wrangler\wrangler-dist\cli.js:141131:13
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async run (C:\Users\bhadoo\AppData\Roaming\npm\node_modules\wrangler\wrangler-dist\cli.js:140737:26) {
  code: 1
}

Node.js v18.16.0

[anaclumos]

Same

FatalError: Failed to upload files. Please try again. Error: {"cause":{"errno":-32,"code":"EPIPE","syscall":"write"}})
    at /Users/spaceship/Downloads/www/node_modules/.pnpm/[email protected]/node_modules/wrangler/wrangler-dist/cli.js:141191:13
    at async run (/Users/spaceship/Downloads/www/node_modules/.pnpm/[email protected]/node_modules/wrangler/wrangler-dist/cli.js:140797:26) {
  code: 1
}

What's weird is that on a particular Wi-fi network, it works flawlessly.

[PBhadoo]

`🌏 Uploading... (166/2249)

C:\Users\bhadoo\AppData\Roaming\npm\node_modules\wrangler\wrangler-dist\cli.js:30632 throw a; ^

FatalError: Failed to upload files. Please try again. Error: {"text":"Received a malformed response from the API","notes":[{"text":"Expired JWT"},{"text":"POST /pages/assets/upload -> 403 Forbidden"}],"kind":"error","name":"ParseError"}) at C:\Users\bhadoo\AppData\Roaming\npm\node_modules\wrangler\wrangler-dist\cli.js:141734:13 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async run (C:\Users\bhadoo\AppData\Roaming\npm\node_modules\wrangler\wrangler-dist\cli.js:141340:26) { code: 1 }`

Wrangler 3.5.1

Can't refresh JWT Token. (happened after 10 minutes maybe?) @sidharthachatterjee Please fix this problem.

[PBhadoo]

@sidharthachatterjee

I guess, you added this https://github.com/cloudflare/workers-sdk/blob/478bed3b80ea353a5be4bd7056b92460a3cf4cd5/packages/wrangler/src/pages/upload.tsx#L320

but this doesn't fix the problem yet this triggers https://github.com/cloudflare/workers-sdk/blob/478bed3b80ea353a5be4bd7056b92460a3cf4cd5/packages/wrangler/src/pages/upload.tsx#L338

Which can be fixed by some method where the error code can be extracted and checked and returned with

jwt = await fetchJwt();
return doUpload();

For me this temporary Fixed Code Worked.

  for (const bucket of buckets) {
    if (bucket.files.length === 0)
      continue;
    attempts = 0;
    const doUpload = /* @__PURE__ */ __name(async () => {
      const payload = await Promise.all(
        bucket.files.map(async (file) => ({
          key: file.hash,
          value: (await (0, import_promises12.readFile)(file.path)).toString("base64"),
          metadata: {
            contentType: file.contentType
          },
          base64: true
        }))
      );
      try {
        logger.debug("POST /pages/assets/upload");
        const res = await fetchResult(`/pages/assets/upload`, {
          method: "POST",
          headers: {
            "Content-Type": "application/json",
            Authorization: `Bearer ${jwt}`
          },
          body: JSON.stringify(payload)
        });
        logger.debug("result:", res);
      } catch (e2) {
        if (attempts < 1000) {
          logger.log("failed:", e2, "retrying... ", attempts);
          await new Promise(
            (resolvePromise) => setTimeout(resolvePromise, Math.pow(2, attempts++) * 1e3)
          );
          if (e2.code === 8000013 || isJwtExpired(jwt)) {
            jwt = await fetchJwt();
          }
          return doUpload();
        } else {
          logger.debug("failed:", e2);
          throw e2;
        }
      }
    }, "doUpload");
    void queue.add(() => {
      return Promise.resolve(doUpload())
        .then(
          () => {
            counter += bucket.files.length;
            rerender(counter, fileMap.size);
          },
          async (error) => {
            logger.log("caught error, retrying with failed problem");
            // Fetch JWT token here
            jwt = await fetchJwt();
            return doUpload();
          /*return Promise.reject(
            new FatalError(
              `Failed to upload files. Please try again. Error: ${JSON.stringify(
                error
              )})`,
              error.code || 1
            )
          );*/
          }
        );
    });
  }

also MAX_UPLOAD_ATTEMPTS is set to 5 is very low. The user should be able send this as parameters. because i uploaded 2000 files and for first 660 files it took around 11 attempts while 10 of them are jwt expire.

I hope this helps.

[thangisme]

I'm also experiencing this problem:

/root/project/node_modules/wrangler/wrangler-dist/cli.js:30947
            throw a;
            ^

FatalError: Failed to upload files. Please try again. Error: {"text":"Received a malformed response from the API","notes":[{"text":"Expired JWT"},{"text":"POST /pages/assets/upload -> 403 Forbidden"}],"kind":"error","name":"ParseError"})
    at /root/project/node_modules/wrangler/wrangler-dist/cli.js:142760:13
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async run (/root/project/node_modules/wrangler/wrangler-dist/cli.js:142289:26) {
  code: 1
}

Node.js v20.7.0

Exited with code exit status 7

[GregBrimble]

Hey folks! We've been incrementally making reliability improvements to our service and to Wrangler, and we're hopefully in a place where any issues with wrangler pages deploy are now very uncommon. If you see consistent failures, please let us know, but I'll close out this issue in the meantime.

[nolanlawson]

I'm running into this same issue with wrangler pages deploy, on a large directory (6k+ files) with several large files (0.5MB max file size). Not sure if any of this information helps, but here's what I'm seeing:

FatalError: Failed to upload files. Please try again. Error: {"text":"Received a malformed response from the API","notes":[{"text":"Expired JWT"},{"text":"POST /pages/assets/upload -> 403 Forbidden"}],"kind":"error","name":"ParseError"})
    at .../tools/image/packages/wrangler/lib/node_modules/wrangler/wrangler-dist/cli.js:140972:13
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async run (.../tools/image/packages/wrangler/lib/node_modules/wrangler/wrangler-dist/cli.js:140496:26) {
  code: 1
}

Node.js v20.9.0

Wrangler version is 3.22.4. I've retried several times, and it always fails after ~15 minutes of trying to upload the files.