"Invalid string length" when merge a large files

[anphanngocvn]

I have a few *.json files (some of them about 750Mb) need to be merged into one *.geofile. However when I run this command to merge them. geojson-merge folder/*.geojson > combined.geojson. I have following errors:

`/geojson-merge/geojson-merge:17 process.stdout.write(JSON.stringify(merge(argv._.map(function(n) { ^

RangeError: Invalid string length at JSON.stringify () at Object. (/geojson-merge/geojson-merge:17:29) at Module._compile (internal/modules/cjs/loader.js:701:30) at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10) at Module.load (internal/modules/cjs/loader.js:600:32) at tryModuleLoad (internal/modules/cjs/loader.js:539:12) at Function.Module._load (internal/modules/cjs/loader.js:531:3) at Function.Module.runMain (internal/modules/cjs/loader.js:754:12) at startup (internal/bootstrap/node.js:283:19) at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)`

How do you tackle this?

[gorbypark]

Hi, I ran into the same problem a few weeks ago. The issue is that you can only add so many arguments to a command (maybe 1000 on bash/linux? I can't remember,). The only solution to this is to do a batch operation (for example, if you have 10,000 files to merge, merge 500 at a time until you have 20 files, and then merge those 20 together). The solution I took was to write a quick node script that iterated over every file in a folder and merges that way. There is not really a max file type restriction in node when iterating over a folder (although you could risk running out of ram!).

[qrcf]

I have the same issue on the entire Kantō region (roads only) of Japan. Its just merging two files. not hundreds. Apparently the file itself is too big at 769.3mb

[warrenca]

@anphanngocvn @qrcf which version of node are you using? i encountered the problem in v6.10, tried in v11 and it's all working fine. I still trying to find out now which node version was it really fixed.

[cagriozkurt]

Might want to remove the 2 from https://github.com/mapbox/geojson-merge/blob/fccdc8d5a53f13f535ee6c538a6edf2d98c57e76/geojson-merge#L19, indentation seems to inflate file sizes.

[jackcjp]

Might want to remove the 2 from

https://github.com/mapbox/geojson-merge/blob/fccdc8d5a53f13f535ee6c538a6edf2d98c57e76/geojson-merge#L19

, indentation seems to inflate file sizes.

Change 2 to 0 works for me! thank you @cagriozkurt