cdloh
Page truncated after ESI processing
I have a test page at https://esi-test-pages.netlify.app/esi-variables.html with the right response header -
surrogate-control: content="ESI/1.0"
And here is my Worker code
import esi from "cloudflare-esi"
export default {
async fetch(request, env, ctx): Promise<Response> {
const url = "https://esi-test-pages.netlify.app/esi-variables.html";
const modifiedRequest = new Request(url, request);
const parser = new esi()
return parser.parse(modifiedRequest)
},
} satisfies ExportedHandler<Env>;
and the output is truncated in the middle for me after the ESI variable tag being processed.
<html lang="en">
<head><title>Test Page for ESI Variables</title></head>
<body>
<p>
Host:
example.myaccount.workers.dev
I am wondering if I did something wrong in my test. Thanks.
👋 Hi @shukitchan!
Your HTML page currently has the esi tag commented out so its a bit hard to test.
I'm surprised it's truncating.
I'll see if I can setup a test page later on to have a play.
Thanks, Callum
The commented piece is like this
<!--esi<esi:vars>$(HTTP_HOST)</esi:vars>-->
So when displayed without ESI processing, it may not show on your page. But if you view the source, you should see the above.
I can remove the <!--esi ... --> if you want. The result to me is the same with the page truncated after the processing.
hey @shukitchan that's working for me. Bit odd. Is your worker returning any logs?
Hmm.. That's strange. I deployed on a Cloudflare account and never run locally, though. Does that make a difference? On the Cloudflare Worker Web UI, I can test my worker running and see the page truncated but there is no logs returned.
@shukitchan @cdloh I had a similar symptom with truncated output when converting from a normal fetch to esi.parse, although based on your code, it seems like the cause may be different.
In my case, the culprit was a subsequent await fetch(apiReq) my worker was making. The truncated output was resolved by moving this to ctx.waitUntil(fetch(apiReq)) (which it should have been anyway).
That said, I still don't fully understand what has happening, or why it only manifest with the esi parser in place. It seemed like what was happening was the subsequent await fetch was short-circuiting worker response (even though I wasn't returning it).
While debugging I also saw lots of TypeError: Illegal invocation: function called with incorrect this reference, which seems likely related.
Are you sure that's your exact worker? I just tested myself in a new worker
<html lang="en">
<head><title>Test Page for ESI Variables</title></head>
<body>
<p>
Host:
<worker-domain>.workers.dev
</p>
</body>
</html>
Yes thats exactly my worker code above. I have no idea why you will have a different result from me, though
I may have to find some time to debug this over the weekend if you have no other ideas.
May be it is a platform issue? I doubt that but i might need to consult with cf as well if i have no luck
I tried 0.4.2 and it is still not working for me.
I debugged this a bit further and this is very strange. The problem is with https://github.com/cdloh/cloudflare-esi/blob/main/src/index.ts#L297
My test ESI page is divided into 3 blocks and for the last block, the "r" variable contains the right value of the last block. And I can print this out with console.log() .
But passing it through line 297 gives me nothing.
I changed line 297 to the following
writer2.write(encoder.encode('<test>' + r + '</test>'));
And I can see the output page with the <test> </test> around each of the 3 block but the last block contains nothing.
Now I think it is a problem of the platform and I should talk to CF instead.
Upon further debugging through a bunch of promises, I find that the above analysis is not right. Instead my problem is fixed if I
- Mark this function as async - https://github.com/cdloh/cloudflare-esi/blob/main/src/index.ts#L306
- add
awaitto https://github.com/cdloh/cloudflare-esi/blob/main/src/index.ts#L308
I think the problem is related to how we should coordinate the writing to the output stream.
Let me know what you think and I can also have submit a PR with the change
The writer does return an async operation. But in all my usage of this in Production it's not been an issue because the data that comes down the wire is chunked anyway. If you can get some reproduction steps together that prove otherwise i'm happy to take MRs @shukitchan
Unfortunately I am not sure how to consistently reproduce this issue on your end. I can do that with my own setup, though.
@cdloh I think I've finally been able to reproduce this!
I'm still working on reproducible test, but in this case, it seems to choke if the page has inline SVGs.
Issues like this could be a reason to use HTMLRewriter to parse things, though I'm yet to use it.
After some more testing, it seems likely this may be large single tags, like those found in SVGs or <img> tags with large srcsets.
@cdloh I was curious, so whipped up an HTMLRewriter version: https://github.com/timkelty/esi-html-rewriter
This is the test that I think cloudflare-esi might choke on: https://github.com/timkelty/esi-html-rewriter/blob/main/tests/long-attributes-and-svg.test.ts
Wow! Thanks heaps for the example etc. I'll see if I can replicate this somewhere in production as well.