import.lua and export.lua corrupt files(devices.lua damages payload data)

[ToMe25]

I originally noticed this while working on #215, however I later noticed that the same issue exists with the current version of devices.lua. Importing or exporting a file using import.lua and export.lua sometimes corrupts them. The imported file will sometimes have a few (2-4 in a 100kB file) bytes replaced with NUL bytes. I primarily tested this using import.lua, however I did also sometimes have this issue with export.lua.

The issue seems to be that cjson.decode sometimes replaces some random numbers with zeros. I found this by printing both the string given to cjson.decode and the result of calling cjson.encode with the new json object to stdout. Then I checked, and the places where there are zeros in the printed string match most of the places where cmp -l finds differences between the damaged file and the original.

Note that this only accounts for most of the issues, there are rare instances where the json object still has the correct byte, and the written file contains a NUL byte. Considering that in #215 simple string copying(well, using string::sub) seems to somehow introduce this issue, I am guessing that the string getting damaged when being copied is what causes these rare cases, but I have no evidence of this.

Also the same file will get corrupted differently when it is corrupted twice while importing, meaning this issue is non-deterministic.

I also verified again that this is an issue with the Lua part of the importing by checking against my import.py script, which never showed this issue in my testing.

I used grep -ao '\[[0-9].*\]' import.log | sed 's/,/\n,/g' | head -<BYTE> | tail +<BYTE> to check the values of the bytes cmp -l showed as changed when printing the messages to the log. This obviously only works if each message is only written once to each file, as otherwise the byte index will be different. <BYTE> being the byte cmp -l shows as changed. Note that cmp -l shows the byte in octal, while the message contains it in decimal, so if they aren't zero but don't match that might be ok.

How to reproduce

1. Create a test file on the host system using dd if=/dev/urandom of=import.txt bs=100K count=1(Bigger files mean a higher chance of this happening)
2. Import said file using import.lua
3. Rename your imported file
4. Import the file again
5. use cmp -l to compare the two files
6. Repeat stop 4 and 5 until cmp -l finds differences, the issue should appear after at most 10 tries

[ToMe25]

I just realized that this is kinda a duplicate of #144. However since this is about the underlying issue in devices.lua, which could cause issues for other programs too, rather than import.lua and export.lua I will leave this open for now. Also this includes a while bunch of info that is probably easier to find this way.