google-takeout-to-sqlite icon indicating copy to clipboard operation
google-takeout-to-sqlite copied to clipboard

Published 20 hours ago • verified publisher icon dogsheep

killed by oomkiller on large location-history

Open khimaros opened this issue 3 years ago • 2 comments

memory seems to grow unbounded and is oom-killed after about 20GB memory usage.

this is happening while loading a ~1GB uncompressed location history.

khimaros avatar Dec 17 '20 00:12 khimaros

The memory profiler tricks I used in https://github.com/dogsheep/healthkit-to-sqlite/issues/7 could help figure out what's going on here.

simonw avatar Dec 17 '20 00:12 simonw

it looks like almost all of the memory consumption is coming from json.load().

another direction here may be to use the new "Semantic Location History" data which is already broken down by year and month.

it also provides much more interesting data, such as estimated address, form of travel, etc.

khimaros avatar Dec 17 '20 00:12 khimaros