jsonpyes icon indicating copy to clipboard operation
jsonpyes copied to clipboard

Published 20 hours ago • verified publisher icon xros

Memory fast filling

Open gsvijayraajaa opened this issue 8 years ago • 3 comments

Hi,

I am trying to index a json file ( about 7GB) with 4 threads. The initial split in data to the threads fast fills the memory and the process gets killed automatically. I am running jsonpyes on AWS m4.xlarge instance.

Regards, Vijay Raajaa GS

gsvijayraajaa avatar Jun 15 '16 09:06 gsvijayraajaa

@gsvijayraajaa

what's the amount of your memory? Have you tried to use 2 threads? And what's the error log?

xros avatar Jul 01 '16 18:07 xros

And verifying data could be very time/memory consuming. You can separate this job by two processes. 1. Verify them somewhere 2. Then import them.

xros avatar Jul 01 '16 18:07 xros

@gsvijayraajaa Maybe now you can update jsonpyes to version 1.3.0 to test it.

https://github.com/xros/jsonpyes/issues/25

For memory safety, it's recommended that make some SWAP room before processing big data.

xros avatar Oct 18 '16 06:10 xros