`text()` does not strip UTF-8 BOM from HTTP response

Open rhargreaves opened this issue 3 years ago • 1 comments

Version: 0.2.4

We have an HTTP backend which adds a UTF-8 BOM to the start of HTTP responses. We noticed that it is present in the string returned by text(). It seems Node's own fetch() method have recently added a fix which strips out BOMs:

https://github.com/node-fetch/node-fetch/pull/1482

Might be worth considering doing something similar? It doesn't feel right that text() returns a string which still contains a BOM. For now, we have made use of https://www.npmjs.com/package/strip-bom to remove the BOM from the string returned by text()

Mar 14 '22 12:03 rhargreaves

It looks like the change would be on this line: https://github.com/fastly/js-compute-runtime/blob/f36dee28789b1187652874fe76d1801c9a8e5974/c-dependencies/js-compute-runtime/js-compute-builtins.cpp#L1138

The consume-body algorithm states that we need to run utf8-decode when the type given is text.

Mar 14 '22 14:03 JakeChampion