geojson-merge

geojson-merge copied to clipboard

I've tried a couple of things to merge ~2k files. First thing I did was merging them on little groups of 100 so I get 20 files of around 30MB each and repeating this process until there is only one file. When I get to the last 4 of them it looks like it tries to take more memory than allowed and stops. FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: node::Abort() [/usr/local/bin/node] 2: node::FatalTryCatch::~FatalTryCatch() [/usr/local/bin/node] 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node] 4: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/local/bin/node] 5: v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [/usr/local/bin/node] 6: v8::String::WriteUtf8(char*, int, int*, int) const [/usr/local/bin/node] 7: node::StringBytes::Write(v8::Isolate*, char*, unsigned long, v8::Local<v8::Value>, node::encoding, int*) [/usr/local/bin/node] 8: node::Buffer::New(v8::Isolate*, v8::Local<v8::String>, node::encoding) [/usr/local/bin/node] 9: node::Buffer::(anonymous namespace)::CreateFromString(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node] 10: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [/usr/local/bin/node] 11: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/usr/local/bin/node] 12: v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node] 13: 0x3a5bea4042fd 14: 0x3a5bea4bd196 sh: line 1: 4733 Abort trap: 6

The other thing I tried is using the -s option so it doesn't limit to the memory available but that gives me another error: (node:4928) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added. Use emitter.setMaxListeners() to increase limit

Are there other ways I can follow to accomplish the same result? My goal is to create an .mbtiles file

You should be able to fix this by adding

NODE_OPTIONS="--max-old-space-size=4096" <command>

--max-old-space-size=4096 measures maximum memory allocation in megabytes - so if you have less ram, decrease the argument as needed.