PMTiles
PMTiles copied to clipboard
protomaps
Consumes all memory on large input
I'm trying to convert a 66 GB file and it quickly consumes 32GB ram then stops.
Is there a solution to converting large files besides launching a super huge VM to handle conversions?
Update: Attempted on a 32 core 248 GB VM and it still gets to a point where it's processing extremely slow. I got to 25 GB fast then after that stalls.
Update 2: Attempted to not compress, file writing stalls around 20GB with 100% CPU usage and 8% memory usage.
The python writer isn't particularly optimized for large use cases, but ideally it should work - that seems like too much ram consumption. What is the total # of tiles in the source (mbtiles?) and are there many duplicates?
I was able to finally have it finish. It gets to a point where nothing is written to disk for minutes at a time. I couldn’t look too far into where that happened.
It’s a planet file which filtered out highways at all zoom levels and smaller roads starting at around zoom 6.
A constant status of what’s happening might be nice since I would tail the output and not get activity in the file for very long times even on a huge machine of 60 cores and 256gb ram.
On Wed, Apr 27, 2022 at 8:45 PM Brandon Liu @.***> wrote:
The python writer isn't particularly optimized for large use cases, but ideally it should work - that seems like too much ram consumption. What is the total # of tiles in the source (mbtiles?) and are there many duplicates?
— Reply to this email directly, view it on GitHub https://github.com/protomaps/PMTiles/issues/43#issuecomment-1111711853, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEP4YQUDZ2N3ZY56ZQWPFLVHICXJANCNFSM5UPRKPPQ . You are receiving this because you authored the thread.Message ID: @.***>
OK, thanks for the report. So it sounds like this isn't a hang. The progress bar might be nice but I don't think there's a clean way to add it to this program without introducing more dependencies. It's likely either collecting deduplication hashes is taking up more and more time (hundreds of millions of hashes) which we could solve by limiting the hash to only tiles smaller than a certain size, or the sorting/pyramid construction takes too much time/ram.
Yeah, I’m not entirely sure where the hang up is but it takes an hour to convert the planet.osm.pbf to mbtiles and longer for mbtiles to pmtiles for larger inputs. Though I was pleased to see it functioning after for the whole planet using go-pmtiles.
On Wed, Apr 27, 2022 at 9:32 PM Brandon Liu @.***> wrote:
OK, thanks for the report. So it sounds like this isn't a hang. The progress bar might be nice but I don't think there's a clean way to add it to this program without introducing more dependencies. It's likely either collecting deduplication hashes is taking up more and more time (hundreds of millions of hashes) which we could solve by limiting the hash to only tiles smaller than a certain size, or the sorting/pyramid construction takes too much time/ram.
— Reply to this email directly, view it on GitHub https://github.com/protomaps/PMTiles/issues/43#issuecomment-1111735278, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEP4YXPFSY7V3WL3CU2SPLVHIIG7ANCNFSM5UPRKPPQ . You are receiving this because you authored the thread.Message ID: @.***>
Go-pmtiles is now the preferred way to convert from mbtiles, which should be much more memory-efficient and shows a progress bar.