itinance
Correct way to write large base64/utf8 files and read them later
Hi, I'm trying to write large lists into storage, upload them to a server, and parse them later in python. These lists will be thousands or even millions of elements long, resulting in files of 100+MB.
When I write them in utf8 format, they can be parsed successfully later on, but if they are longer than a few thousand elements (~30 MB), the app hangs when writing the file, and the file never appears at the path. If they are short, the process works fine.
When I write them in base64 format, they are successfully written very quickly. Even huge files near 1 gb are successfully written to storage with this format. However, when I try to parse them later on, they are always corrupted. I've tried everything from standard stuff like: https://kite.com/python/examples/3422/base64-decode-a-%60base64%60-file to online base64 decoder services, so I'm fairly certain I'm not decoding it wrong.
Does anyone have a suggestion for how to
- Write large utf8 files so the app doesn't crash OR
- Write base64 files correctly so that they can be decoded later
My react native code is as follows:
var list = [1,1,1,....1,2] // can be 100000+ elements
var list2 = [2,2,2,...1,2] // can be 100000+ elements
var json = JSON.stringify({x: list, y: list2})
var path = RNFS.ExternalDirectoryPath + '/test.json';
//alert(JSON.stringify({'x': xlist, 'y': ylist, 'z': zlist}))
// write the file
let format = 'utf8' //OR 'base64'
RNFS.writeFile(path, json, format)
.then(async (success) => {
})
.catch((err) => {
});
I'm having a similar issue when writing audio files. The promise on writeFile is fulfilled but no file is created and no error is caught. Pretty hopeless for debugging...
