fast-text-encoding icon indicating copy to clipboard operation
fast-text-encoding copied to clipboard

Could you guys please support utf-16le

Open chiro-hiro opened this issue 3 years ago • 2 comments

Some packages that's depend on this package won't work on Node.js

RangeError: Failed to construct 'TextDecoder': The encoding label provided ('utf-16le') is invalid.
    at new k (.../node_modules/fast-text-encoding/text.min.js:1:134)
    at Object.<anonymous> (.../node_modules/rustbn.js/lib/index.asm.js:1:17613)

The reason is fast-text-encoding package didn't support utf-16le

chiro-hiro avatar Jan 27 '21 06:01 chiro-hiro

From Node 5.1 (~2015), this polyfill uses Buffer.toString to decode the buffer, but only allows UTF-8. And from 11.0 (~2018) it's not needed at all as Node supports TextEncoder and TextDecoder.

I suppose this could instead pass through the encoding to the Buffer.toString call and let Node's built-in behavior take over, but it would be a bunch of work for a specific benefit. If the package that uses this one needs 'utf-16le' directly why not modify it to use Buffer directly?

samthor avatar Jan 27 '21 07:01 samthor

Hi @samthor,

Thank for your reply, this package provide very basic function and dependent package were use it careless. They've just passed utf-16le regardless environment. Clearly It isn't your fault but no one going to fix this since they think it wasn't their fault too.

In this case, I think lib/asm.js must take responsibility, I'm not sure do they still maintenance it.

var UTF16Decoder = typeof TextDecoder !== "undefined" ? new TextDecoder("utf-16le") : undefined;

chiro-hiro avatar Jan 27 '21 07:01 chiro-hiro

@samthor I have the same issue! is it possible to support utf-16le?

RangeError: Failed to construct 'TextDecoder': The encoding label provided ('utf-16le') is invalid.

samsaurio avatar Aug 18 '22 15:08 samsaurio

I've just published v1.0.6, which more aggressively falls through to Buffer if it's available. This fixes this for Node v5.1 => v11. (From v11, TextDecoder is already supported natively`).

Also, please upgrade to the latest LTS. This polyfill should not exist 🤣

samthor avatar Aug 30 '22 22:08 samthor