readAsText function causes encoding error reading large files

[chauthai]

cordova-file-plugin encoding error bug report

Abstract

The readAsText function of the cordova-file-plugin causes an encoding error when reading an UTF-8 file larger than 2 megabytes on platforms iOS, Android and Windows.

Description

The error is caused by the native implementation of the readAsText function of the cordova-file-plugin. Files are cut into chunks at a predefined size and are immediately converted to UTF-8. UTF-8 encoded characters use 4 bytes and if the cut is exactly in one character with 4 bytes, an encoding error is thrown. You can see the UTF-8 leading byte e2 at the end of data-chunk.dump file. The missing bytes are in the next chunk and if you convert the dump file back to text a filler character is shown (e.g. a ?).

Workaround

We implemented a workaround using the readAsArrayBuffer function of the cordova-file-plugin and converting the typed array to UTF-8 in JavaScript with TextDecoder.decode().

How to reproduce the encoding error

1. Check out sampe-app
2. Install dependencies npm i
3. Initialize Cordova npm run init
4. Deploy on platform npm run ios/android/windows
5. Press Fire Parser in Demo App
6. Watch the logs
7. Set a breakpoint at the throw of encoding error in iOS
8. Look at the data variable

Related to #238

[chauthai]

@janpio Any updates on this issue? The bug also occurs on the UWP platform.

[NateEag]

Just wanted to add that I ran into this problem in a project at my job.

It's not that hard to encounter it - loading a file with more than 262,144 bytes that's primarily composed of multibyte characters (that is, non-ASCII ones, in UTF-8 encoding) should do it often.

Here is a bit of Python 3 that generates a UTF-8 file that should consistently break the readAsText() function:

longstring = 'a' * 262143 + u'‘abc'
file = open('testfile.txt', 'w')
file.write(longstring)
file.close()

[nip3o]

This happened to me today as well, but only on iOS. Looking at the implementation I don't see how the same issue could occur on Android however – in the source code at FileUtils.java#L1086 it seems like the whole result is written to the buffer before it gets converted into a string.

I haven't tested it in depth on Android though, so I might be missing something.

[WilsonFpz]

Same issue here. Happened randomly in IOS only. Please help fix it. thanks.

[YvesAmmann]

+1

[c0dem4ster]

+1

[NateEag]

For the +1s, some details would be nice.

In my case, the workaround suggested in the report worked just fine.

Have you tried it? If so, did it solve your problem?

[YvesAmmann]

@NateEag: Yes, the workaround with readAsArrayBuffer did worked for me – but it’s only a workaround, not a fix. :)

[NateEag]

Gotcha. Yes, a real fix would obviously be better. I just wanted to be sure there wasn't some subtlety being missed that meant the workaround wasn't fully functional.

Thanks for the details, @YvesAmmann !

[MauroIT]

Anyone found a workaround for the Windows platform?

[NateEag]

@MauroIT Does the workaround suggested above fail on Windows? If so, how does it fail?

[MauroIT]

I've added a Polyfill for window.TextDecoder. It works on Windows now :)

[webzfactory]

+1 Thank you for the work around, but this issue should really be addressed.

[ouou12138]

have you try with 30m+ file? I got an error like this when i try a 30m+ large file：

{"type":"error","bubbles":false,"cancelBubble":false,"cancelable":false,"lengthComputable":false,"loaded":0,"total":0,"target":{"_readyState":2,"_error":{"code":1},"_result":null,"_progress":0,"_localURL":"http://localhost/__cdvfile_sdcard__/Download/yuanshen_4.0.0.apk","_realReader":{}}}