jimhigson
Handling large JSON file
Hello! I've been looking for something like oboe! I have json files that contain geometry to be rendered in Three.js (a javascript webgl library). With Oboe, I should be able to start drawing this geometry while still loading the file (eventually!). The reason I found Oboe was because I have a user that was trying to load a rather large JSON file. My users can run my Javascript app locally and on the web. This particular JSON file is 295.8mb. While I have been able to load 100mb files just fine, I was getting allocation size overflow errors in Firefox, and Unexpected end of input errors in Chrome with this 295.8mb file.
So I started looking at oboe, and with my other files, I get the expected behavior. I can use the .node syntax to notify me when these particular elements are being loaded. This large file has a rather large chunk at the beginning (two gigantic arrays). After 3 minutes in Chrome, I finally reach the .fail callback for oboe and I never hit any of the .node elements as I do with smaller json files.
I'm obviously hitting a limit here, but I'm not sure what is imposing this limit. Is it the browser? Is it the available RAM on the system (16GB)? Is it the XMLHttpRequest object? Is there a way with oboe to get more useful information about the failure other than what the browser is giving me?
Any help is appreciated.
Not that I know of, but as far as helping you load that data, have you looked at oboe.drop? In this repo I successfully loaded around 180mb https://github.com/JuanCaicedo/oboe-memory-bug
Hi @JuanCaicedo, Thank you for this example. Unfortunately, Chrome chokes at 152mb of my data (I get the Aw snap! page at /home?drop=true). The example with your data behaves as you explain (fails at /home, works at /home?drop=true). I tried another file, this time 27.9mb and both /home and /home?drop=true were able to load it.
Is there any way you can truncate the data? Maybe you could start it at 150mb and try progressively larger files
How are you able to tell it loaded 152 mb, through Chrome dev tools?
@juancaicedo, Yes, through dev tools it shows the loaded amount.
I'll try what you say and see if I discover anything as the files get bigger.
You guys come to any conclusions? I'm about to go down this same path of using Oboe with Threejs.
I have a idea of gzip/zip the raw JSON, then stream the zipped json. 300MB.json=50MB.zip, 100MB.json=20MB.zip, It will make 5 times faster.
I have identical problem. Reading a 500M json file using oboe.js will fail, even using oboe.drop.
My code is like: function testOboe() { oboe('result.json') .node('!.*', function(myObject){ document.writeln(myObject["globalId"]); return oboe.drop; }).done(function(finalJson){ document.writeln("load complete!"); }) }
What's the problem?
