UTF-16 compression seems to be broken on Safari

[auromota]

Hi there! First of all, this library is amazing. Thanks. The results were great.

I started using this library few days ago and have been testing it mainly on Chrome/Windows. I was using the UTF-16 compression method. It worked great. However, on latest Safari 10/Sierra, the method seems to be broken, though no errors are thrown. So I've tested the compress method and it also fails in my app. The only way I could get it working was to use the base64 method. Since then, it's been working on all browsers I could test.

I'm not sure what's the problem as I've not done any deeper investigation.

[christianrondeau]

Hey! We had a similar problem. In fact, what's happening is that UTF-16 gets "converted" to UTF-8, which keeps the "string" as is, but will break some of the actual bytes once transferred by HTTP.

There's no clean solution but to give up on UTF16 encoding, and instead use Base64 or EncodedURIComponent.

[JobLeonard]

@auromota, was there a place in the documentation where you expected to see this caveat mentioned where it was not? If so we'll keep the issue open until that is fixed, otherwise I guess we can close this, right?

[pieroxy]

Well, compressToUTF16 is supposed to work on all browsers. At least according to http://pieroxy.net/blog/pages/lz-string/index.html#inline_menu_6

[silk-bahamut]

Same here, decompressFromUTF16 and compressToUTF16 do not work on safari or IE 11

[squidfunk]

We had a similar problem. In fact, what's happening is that UTF-16 gets "converted" to UTF-8, which keeps the "string" as is, but will break some of the actual bytes once transferred by HTTP.

If you're sending compressed data (doesn't matter if you use compress or compressToUTF16) via XMLHTTPRequest, you have to pass the data to XMLHTTPRequest.send as a Uint8Array and it will work as expected - it's basically like an asynchronous file upload which works in all browsers.

Using a simple Express middleware in Node.js as a backend, the difference is easily spotable:

Sending data as a Uint8Array:

[ <Buffer 00 04 00 40 0c 00 20 0a 00 60 07 00 04 01 20 05 00 68 00 c0 16 00 70 03 c0 00 80 22 00 48 03 20 02 80 2a 00 68 03 a0 01 80 26 00 58 03 60 03 80 2e 00 ... > ]

Sending data as a string, browsers will remove all non-printable characters (e.g. \u0000) from the body, effectively corrupting the data:

[ <Buffer 04 40 e0 b0 80 e2 80 8a 60 dc 80 d0 81 e2 80 85 68 c3 80 e1 98 80 e7 80 83 ec 80 80 e8 80 a2 48 cc a0 ca 80 e2 a8 80 e6 a0 83 ea 80 81 e8 80 a6 58 cd ... > ]

[tobspr]

Any workarround for this? Facing the same issue, but I can't send UInt8Arrays since I am sending the encoded strings as JSON. Maybe an utf16 version without non-printable characters?

[squidfunk]

From my testing only sending data as binary worked consistently well. You could use base64 encoding but this may rip the benefits of compression.

[pieroxy]

You are probably using the wrong library if you want your results in a UInt8Array. The whole point of LZ-String is to produce "packed" string, since that's what you store in localStorage, and browsers store it as UTF-16. In order to put a compressed string in JSON, you may use compressToBase64. That is guaranteed to stay untouched by any conversion of ASCII, Unicode or anything else. Of course, the compression ratio is much worse.

[pieroxy]

Same here, decompressFromUTF16 and compressToUTF16 do not work on safari or IE 11

@silk-bahamut Can you provide a working page that demonstrate the issue?

Most of the time the issue will be conversions between UTF16 to UTF8 and back to UTF16. This is not a binary-conservative conversion. In other words, you will get back a string that is not the same than the one you began with. Obviously LZ-String cannot decompress it successfully.

LZ-String is meant to compress strings into other strings to be stored in localStorage, which uses UTF-16, so there is no conversion.

[silk-bahamut]

@pieroxy sorry it was 4 years ago, I do not have access to the project anymore

[Rycochet]

There are now support functions for binary safe conversion to/from uint8array (useful for network transport as well) - have a look at https://github.com/pieroxy/lz-string/blob/master/src/Uint8Array/utils.ts (bear in mind that v2 is still in development and not released to npm yet!)