cloudflare
🐛 Bug Report — TypeError: memory limit has been exceeded with inline `data:` URL
What is happening
When serving an HTML file with a large (>10MB) inline data: URL resource, even a basic worker will halt the response and respond with:
TypeError: Parser error: The memory limit has been exceeded.
I thought this was simply the response size, but this is not the case.
- If I send the file as
text/plain, the file will send through the worker - If I generate a large HTML file with
/dev/random, the file will send through the worker - The response truncates right at the point in the file that a data URL starts
So, it does seem to me to be very specific to inline data URLs.
Does this imply the worker fetch() is somehow inspecting the data URL during streaming? This seems odd, but if so, is there a missing option to disable this?
Reproducing
- Run a minimal worker
- Output an HTML file with a large inline data resource
- Try to fetch the file through the worker
Example data URL HTML generator:
#!/bin/sh
content=$(dd if=/dev/random bs=1MB count=10 | base64 -w0)
echo "<html><head></head><body><img src='data:image/png;base64,$content' /></body></html>"
Generate the HTML file and serve it:
% ./gen.sh > 10mb.html
% python -m http.server 9000
And the worker:
export default {
async fetch(request, env, ctx) {
const response = await fetch("http://localhost:9000/10mb.html");
return response;
}
}
Run and try to fetch:
% wrangler dev test.js
% curl -o - http://localhost:8787 > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 25 0 25 0 0 46 0 --:--:-- --:--:-- --:--:-- 46
Notice only 25 bytes sent, which is the start of the img element
% dd if=10mb.html bs=25 count=1
<html><head></head><body>
I'm unable to reproduce any failure here with workerd given the example included. The error that you mention ("Parser error: The memory limit has been exceeded") is an error that HTMLRewriter would throw (it comes from lol-html). Your example is not using HTMLRewriter, however so I'm a bit confused. Can you please clarify?
Also, are you seeing this error with a production worker (e.g. wrangler deploy, run the worker) or in local dev (e.g. wrangler dev), etc.
This has me incredibly confused as well, there is indeed no instance of HTMLRewriter. What I posted as the worker script is what I am testing with, in a fresh/empty directory too, and running the wrangler dev command I noted above.
The exception is from the html-rewriter code though:
✘ [ERROR] workerd/server/server.c++:3422: error: Uncaught exception: workerd/api/html-rewriter.c++:99: failed: remote.jsg.TypeError: Parser error: The memory limit has been exceeded.
A couple more pieces:
% npm list
@[email protected] /tmp/test
├── @cloudflare/[email protected]
├── [email protected]
├── [email protected]
├── [email protected]
└── [email protected]
% node --version
v18.7.0
Also, I notice that the first request will succeed with a bad response (no error but only 25 bytes sent) and the second request will fail with a bad response (error raised and only 25 bytes sent).
I can only follow along loosely with C++, but notice there is implicit parsing of data URLs outside the HTML rewriter:
https://github.com/cloudflare/workerd/blob/a9cf631a97194d0c2d6d46c281d874b1c5665fc8/src/workerd/api/data-url.c%2B%2B#L17
It seems like there is some processing of data URLs as sub requests inside a fetch() call?
The DataUrl implementation is entirely separate from the HTMLRewriter and we do not automatically parse data urls that are embedded in a stream. The only place a data: URL is handled is if you are explicitly passing that to a fetch call, e.g. await fetch('data:...'). There's nowhere else in the code where it is used directly.
Yeah I saw that parsing of the fetch() URL was covered there, I didn't see any tests describing what I'm noticing in the worker example above though.
When you say you aren't able to reproduce this bug, what do you notice on the response? Are you getting 25 bytes back, or the full 10MB?
Would a docker compose or something help verify this bug? I could stumble through rebuilding workerd too, there might be some way to catch what is happening with the data URL? I was sure that the minimal repro steps here would surface this bug for you, but I'm rather confused all around now.
When you say you aren't able to reproduce this bug, what do you notice on the response? Are you getting 25 bytes back, or the full 10MB?
Keeping in mind that I'm testing with workerd only (not wrangler dev) ... I get the full response.
I don't know enough about using workerd directly, if there is something I can run locally and confirm what you're seeing, let me know.
Is it possible this is specific to wrangler/workers-sdk otherwise?
It's possible since in that path something may be injecting polyfills and maybe HTMLRewriter use into the worker. Take a look through the compiled source that the CLI generates and see if you can spot anywhere HTMLRewriter may be getting added.
Are you describing generating this with Wrangler? With something like:
% wrangler deploy --dry-run --outdir dist
When I do that, the bundled file looks like this:
// test.js
var test_default = {
async fetch(request, env, ctx) {
const response = await fetch("http://localhost:9000/10mb.html");
return response;
}
};
export {
test_default as default
};
//# sourceMappingURL=test.js.map
I don't know exactly what to generate here, so here's a docker compose with the minimal reproduction. This is just the example above, but isolated via containers to prove that there is indeed no HTMLWriter defined at any point in the worker.
https://github.com/agjohnson/wrangler-large-data-url
Let me know if any part of that isn't clear. If this seems like a wrangler issue, could we transfer the issue?
I've updated my test case above with wrangler 3.102.0 and the bug still exists.
How can I help here, is this the wrong repository to be discussing this bug?
I think it's the wrong repo, yeah. This occurs in the dev server, not the workers runtime:
https://github.com/cloudflare/workers-sdk/blob/135dd8537642b756a4018b256d125e742995e5d9/packages/wrangler/templates/startDevWorker/ProxyWorker.ts#L252
If the worker's response is HTML, then it uses a rewriter to add things like a live reload script. But this is a post-processing step on the response and does not modify your worker's code.
To clarify more, this is just a local reproduction of bug we are hitting in production Cloudflare workers. The actual issue we're having is not with the dev server.
If the worker's response is HTML, then it uses a rewriter to add things like a live reload script.
What about outside the dev server, in production workers? We first triggered this bug in production simply by including an <img> element with a large data: URI in HTML source. We aren't doing any actual handling or rewriting of data URIs though, there is something implicit happening at the worker level to process data URIs.
I've updated my compose example to use [email protected], the bug is still present there. This issue hasn't been transferred yet so I created a new issue on worker-sdk to continue trying to resolve this bug.