wahooMapsCreator icon indicating copy to clipboard operation
wahooMapsCreator copied to clipboard
verified publisher icon treee111

Java heap space - out of memory

Open FrenzyNatra opened this issue 3 years ago • 8 comments

Expected Behavior

Map files get created.

Current Behavior

Java heap space error.

Steps to Reproduce the Issue

  1. Map creation started via GUI (wahoomc 2.0.2)

Context

  • OS/Platform: Win10 x64
  • Other information regarding your environment (optional): Java 1.8.0_341

Log Output / Stack Trace

{...}

FrenzyNatra avatar Oct 07 '22 19:10 FrenzyNatra

Hi @FrenzyNatra! for which country are you creating maps for? What happens if you install alpha15 and try creating the map again? Install that alpha15 via pip install wahoomc==2.1.0a15

treee111 avatar Oct 09 '22 21:10 treee111

Hey @treee111 Tried to create maps for Malta ✅ Germany incl. border countries ❌ Germany w/o border countries ❌

I'll try again using 2.1.0a15 and get back to you ✌️. Thanks.

FrenzyNatra avatar Oct 10 '22 07:10 FrenzyNatra

Thanks for reporting back! So it works basically and the installation went well. Then it gets kind of difficult l - if even possible - to develop a solution. This is most probably related to your hardware. Working with a big map file (Germany) is too much for the hardware.

Could you include the whole output in CMD when this error happens? calling via CMD might output more? python -m wahoomc cli -co germany

I'll ask a expert for processing under Windows for advice!

treee111 avatar Oct 11 '22 06:10 treee111

As this is a memory problem, closing any other apps before starting wahoomapsCreator would be a first step.

Some more input would be good for this issue:

  • Hardware specifications: CPU, RAM, ...
  • what happens to RAM in Task Manager when you process and it crashes?
  • Can you see RAM usage constantly increasing?

Thanks!

treee111 avatar Oct 11 '22 08:10 treee111

Hey @treee111 , thanks for developing such a cool thing and sorry for the poor description in the first post.

System: CPU: Intel(R) Core(TM) i7-8650U RAM: 16 GB OS: Win10 Pro

1st/2nd try

  • started via python -m wahoomc gui (germany, defaults) or CLI python -m wahoomc cli -co germany (v2.1.0a15)
  • CPU/RAM: ~30% / 70-80% (total)
  • no DEBUG output (compared to previous version)
  • manually cancelled after ~30min of inactivity (see last log line)

3rd try

  • cleared a mess of Chrome tabs (~ 2-3 GB)
  • started via CLI python -m wahoomc cli -co germany
  • CPU (total / wmc-process): ~ 30-45 % / 10-25 %
  • RAM (total / wmc-process): ~ 70 % / 20-40 %

So, since the upgrade there was no error output anymore, but the process always seemes to halt (no CPU activity) after "sorting land* osm files: OK". RAM usage seems to be quite steady, so no significant increase over time. I am a bit wondering why the previous step (merging tiles) stops before all tiles are processed (see log). Besides that, every time executing wahoomc, one more tile seems to get merged. Most likely there is an error not being prompted. Is there a chance to get verbose log output?

Update: Tried again after removing folder "wahooMapsCreatorData/_tiles" (avoid risk of potentially corrupted files from previous runs) but got Error in OSMConvert with country: {key}. In ".../maps/" there are two pbf files for czech republic ("czech_republic[...]": 773 MB and "czech-republic[...]": 232 MB). Don't think this is related to the primary issue, but maybe still worth mentioning. Next I'll clear the whole data folder to force processing from scratch....

~~~ Logs 
1st try:
    Uninstalling wahoomc-2.0.2:
      Successfully uninstalled wahoomc-2.0.2
Successfully installed requests-2.28.1 wahoomc-2.1.0a15
(gdal-user) C:\Users\[***]\Downloads\wahoomc>python -m wahoomc gui
INFO:--------------------------------------------------------------------------------
INFO:# Input country: germany.
INFO:--------------------------------------------------------------------------------
INFO:+ Involved country: germany
INFO:--------------------------------------------------------------------------------
INFO:# check C:\Users\[***]\wahooMapsCreatorData\_download\land-polygons-split-4326\land_polygons.shp file
INFO:--------------------------------------------------------------------------------
INFO:# check countries .osm.pbf files
INFO:+ Checking for old maps and remove them
INFO:+ mapfile for germany: up-to-date. Input: germany.
INFO:--------------------------------------------------------------------------------
INFO:# Filter tags from country osm.pbf files
INFO:+ Filter tags from country osm.pbf files: OK
INFO:--------------------------------------------------------------------------------
INFO:# Generate land
INFO:+ Generate land: OK
INFO:--------------------------------------------------------------------------------
INFO:# Generate sea
INFO:+ Generate sea: OK
INFO:--------------------------------------------------------------------------------
INFO:# Split filtered country files to tiles
INFO:+ Splitting tile 1 of 57 for Coordinates: 137,88 from map of germany
INFO:+ Splitting tile 2 of 57 for Coordinates: 134,89 from map of germany
INFO:+ Splitting tile 3 of 57 for Coordinates: 135,89 from map of germany
INFO:+ Splitting tile 4 of 57 for Coordinates: 136,89 from map of germany
[...]
INFO:+ Splitting tile 55 of 57 for Coordinates: 137,81 from map of germany
INFO:+ Splitting tile 56 of 57 for Coordinates: 136,85 from map of germany
INFO:+ Splitting tile 57 of 57 for Coordinates: 133,81 from map of germany
INFO:+ Split filtered country files to tiles: OK
INFO:--------------------------------------------------------------------------------
INFO:# Merge splitted tiles with land an sea
INFO:+ Merging tiles for tile 1 of 57 for Coordinates: 137,88
INFO:+ Merging tiles for tile 2 of 57 for Coordinates: 134,89
INFO:+ Merging tiles for tile 3 of 57 for Coordinates: 135,89
INFO:--------------------------------------------------------------------------------
INFO:# Sorting land* osm files
INFO:+ Sorting land* osm files: OK

3rd try:
[...]
INFO:--------------------------------------------------------------------------------
INFO:# Merge splitted tiles with land an sea
INFO:+ Merging tiles for tile 1 of 57 for Coordinates: 137,88
INFO:+ Merging tiles for tile 2 of 57 for Coordinates: 134,89
INFO:+ Merging tiles for tile 3 of 57 for Coordinates: 135,89
INFO:+ Merging tiles for tile 4 of 57 for Coordinates: 136,89
INFO:+ Merging tiles for tile 5 of 57 for Coordinates: 137,89
INFO:+ Merging tiles for tile 6 of 57 for Coordinates: 132,85
INFO:--------------------------------------------------------------------------------
INFO:# Sorting land* osm files
INFO:+ Sorting land* osm files: OK

After removing folder ".../_tiles/":
(gdal-user) C:\Users\[***]>python -m wahoomc cli -co germany
INFO:--------------------------------------------------------------------------------
INFO:# Input country: germany.
INFO:--------------------------------------------------------------------------------
INFO:# Determine involved/border countries
INFO:+ Involved country: czech_republic
INFO:+ Involved country: germany
INFO:+ Involved country: austria
INFO:+ Involved country: liechtenstein
INFO:+ Involved country: switzerland
INFO:+ Involved country: italy
INFO:+ Involved country: netherlands
INFO:+ Involved country: belgium
INFO:+ Involved country: luxembourg
INFO:+ Involved country: france
INFO:+ Involved country: poland
INFO:+ Involved country: denmark
INFO:+ Border countries will be processed
INFO:--------------------------------------------------------------------------------
INFO:# check C:\Users\[***]\wahooMapsCreatorData\_download\land-polygons-split-4326\land_polygons.shp file
INFO:--------------------------------------------------------------------------------
INFO:# check countries .osm.pbf files
INFO:+ Checking for old maps and remove them
INFO:+ mapfile for czech-republic: up-to-date. Input: czech_republic.
INFO:+ mapfile for germany: up-to-date. Input: germany.
INFO:+ mapfile for austria: up-to-date. Input: austria.
INFO:+ mapfile for liechtenstein: up-to-date. Input: liechtenstein.
INFO:+ mapfile for switzerland: up-to-date. Input: switzerland.
INFO:+ mapfile for italy: up-to-date. Input: italy.
INFO:+ mapfile for netherlands: up-to-date. Input: netherlands.
INFO:+ mapfile for belgium: up-to-date. Input: belgium.
INFO:+ mapfile for luxembourg: up-to-date. Input: luxembourg.
INFO:+ mapfile for france: up-to-date. Input: france.
INFO:+ mapfile for poland: up-to-date. Input: poland.
INFO:+ mapfile for denmark: up-to-date. Input: denmark.
INFO:--------------------------------------------------------------------------------
INFO:# Filter tags from country osm.pbf files
INFO:+ Converting map of czech_republic to o5m format
ERROR:! Error in OSMConvert with country: {key}

FrenzyNatra avatar Oct 11 '22 21:10 FrenzyNatra

Hi @FrenzyNatra, thanks for the detailled description of the different steps! That helps a lot 👍

The version 2.1.0a15 is way further than the 2.0.2. Because of that some things are different: debug logging, different naming of some countries, ...

Prerequisite for further runs:

  • I would like you to first delete the whole wahooMapsCreatorData/download/maps and wahooMapsCreatorData/_tiles directory, like you posted.
  • I also have both czech files but with a similar size. But better to clean up

For 1st and 2nd:

  • I enabled debug output and added some debug logging in the location where the error seams to occur.
  • You are invited to install 2.1.0a17 and post what is outputted.

For 3rd:

  • see prerequisite
  • I also added debug output in that location in 2.1.0a17.

You could also add the arguments -fp to force all processing or -fd to force downloading the maps files or use checkboxes via GUI. If a DL was interrupted, a maps-file could be corrupt.

Happy to hear from you, thanks again!

treee111 avatar Oct 17 '22 21:10 treee111

Yeah, you're welcome! And thanks for the detailed feedback!

I'd suggest to make the deletion of output files a standard procedure in case of suspicious errors for all further runs (label "empty-output"). In the latest run, where files in wahooMapsCreatorData/download were removed prior to execution, the OSMConvert error didn't show up again 🙌

FYI: in the meanwhile I have setup the dev-environment in VS Code. Don't know if that might have side-effects on the following test runs (e.g. initialized conda for powershell, etc.). Struggled with some modules, but that's a different story...

Summary for 2.1.0a17: Run 1:

  • Map creation Malta (empty-output): ✔️

Run 2:

  • Map creation Germany: ❌ (see logs)
~~~~~~~~~ Log output ~~~~~~~~~~~~~~ 
~~ Run 2 ~~~~~~~~~~~~~~~~~~~~~~~~

INFO:+ Split filtered country files to tiles: OK
INFO:--------------------------------------------------------------------------------
INFO:# Merge splitted tiles with land an sea
INFO:+ Merging tiles for tile 1 of 57 for Coordinates: 137,88
INFO:--------------------------------------------------------------------------------
INFO:# Sorting land* osm files
DEBUG:subprocess:'Oct 19, 2022 11:08:04 PM org.openstreetmap.osmosis.core.Osmosis run'
DEBUG:subprocess:'INFO: Osmosis Version 0.48.3'
DEBUG:subprocess:'Oct 19, 2022 11:08:05 PM org.openstreetmap.osmosis.core.Osmosis run'
DEBUG:subprocess:'INFO: Preparing pipeline.'
DEBUG:subprocess:'Oct 19, 2022 11:08:05 PM org.openstreetmap.osmosis.core.Osmosis run'
DEBUG:subprocess:'INFO: Launching pipeline execution.'
DEBUG:subprocess:'Oct 19, 2022 11:08:05 PM org.openstreetmap.osmosis.core.Osmosis run'
DEBUG:subprocess:'INFO: Pipeline executing, waiting for completion.'
DEBUG:subprocess:'Oct 19, 2022 11:08:06 PM org.openstreetmap.osmosis.core.Osmosis run'
DEBUG:subprocess:'INFO: Pipeline complete.'
DEBUG:subprocess:'Oct 19, 2022 11:08:06 PM org.openstreetmap.osmosis.core.Osmosis run'
DEBUG:subprocess:'INFO: Total execution time: 1203 milliseconds.'
INFO:+ Sorting land* osm files: OK
DEBUG:osmosis_win_file_path: C:\Users\[***]\Anaconda3\envs\gdal-user\lib\site-packages\wahoomc\tooling_win\Osmosis\bin\osmosis.bat
DEBUG:cmd: ['C:\\Users\\[***]\\Anaconda3\\envs\\gdal-user\\lib\\site-packages\\wahoomc\\tooling_win\\Osmosis\\bin\\osmosis.bat', '--rbf', 'C:\\Users\\[***]\\wahooMapsCreatorData\\_tiles\\137\\88\\split-czech_republic.osm.pbf', 'workers=1', '--rbf', 'C:\\Users\\[***]\\wahooMapsCreatorData\\_tiles\\137\\88\\split-czech_republic-names.osm.pbf', 'workers=1', '--merge', '--rbf', 'C:\\Users\\[***]\\wahooMapsCreatorData\\_tiles\\137\\88\\split-germany.osm.pbf', 'workers=1', '--merge', '--rbf', 'C:\\Users\\[***]\\wahooMapsCreatorData\\_tiles\\137\\88\\split-germany-names.osm.pbf', 'workers=1', '--merge', '--rbf', 'C:\\Users\\[***]\\wahooMapsCreatorData\\_tiles\\137\\88\\split-austria.osm.pbf', 'workers=1', '--merge', '--rbf', 'C:\\Users\\[***]\\wahooMapsCreatorData\\_tiles\\137\\88\\split-austria-names.osm.pbf', 'workers=1', '--merge', '--rx', 'file=C:\\Users\\[***]\\wahooMapsCreatorData\\_tiles\\137\\88\\land1.osm', '--s', '--m', '--rx', 'file=C:\\Users\\[***]\\wahooMapsCreatorData\\_tiles\\137\\88\\sea.osm', '--s', '--m', '--tag-transform', 'file=C:\\Users\\[***]\\Anaconda3\\envs\\gdal-user\\lib\\site-packages\\wahoomc\\resources\\tunnel-transform.xml', '--wb', 'C:\\Users\\[***]\\wahooMapsCreatorData\\_tiles\\137\\88\\merged.osm.pbf', 'omitmetadata=true']
DEBUG:before run_subprocess_and_log_output
DEBUG:subprocess:'Oct 19, 2022 11:08:06 PM org.openstreetmap.osmosis.core.Osmosis run'
DEBUG:subprocess:'INFO: Osmosis Version 0.48.3'
DEBUG:subprocess:'Oct 19, 2022 11:08:06 PM org.openstreetmap.osmosis.core.Osmosis run'
DEBUG:subprocess:'INFO: Preparing pipeline.'
DEBUG:subprocess:'Oct 19, 2022 11:08:06 PM org.openstreetmap.osmosis.tagtransform.impl.TransformLoader parseTranslation'
DEBUG:subprocess:'INFO: New translation: Adjust tunnel Layers'
DEBUG:subprocess:'Oct 19, 2022 11:08:06 PM org.openstreetmap.osmosis.core.Osmosis run'
DEBUG:subprocess:'INFO: Launching pipeline execution.'
DEBUG:subprocess:'Oct 19, 2022 11:08:06 PM org.openstreetmap.osmosis.core.Osmosis run'
DEBUG:subprocess:'INFO: Pipeline executing, waiting for completion.'
DEBUG:subprocess:'Exception in thread "pool-5-thread-1" java.lang.OutOfMemoryError: Java heap space'
DEBUG:subprocess:'at com.google.protobuf.LongArrayList.addLong(LongArrayList.java:199)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$DenseNodes.<init>(Osmformat.java:12641)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$DenseNodes.<init>(Osmformat.java:12578)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$DenseNodes$1.parsePartialFrom(Osmformat.java:14081)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$DenseNodes$1.parsePartialFrom(Osmformat.java:14075)'
DEBUG:subprocess:'at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:888)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveGroup.<init>(Osmformat.java:4635)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveGroup.<init>(Osmformat.java:4574)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveGroup$1.parsePartialFrom(Osmformat.java:6604)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveGroup$1.parsePartialFrom(Osmformat.java:6598)'
DEBUG:subprocess:'at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:888)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveBlock.<init>(Osmformat.java:3113)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveBlock.<init>(Osmformat.java:3048)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveBlock$1.parsePartialFrom(Osmformat.java:4432)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveBlock$1.parsePartialFrom(Osmformat.java:4426)'
DEBUG:subprocess:'at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:158)'
DEBUG:subprocess:'at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:191)'
DEBUG:subprocess:'at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:203)'
DEBUG:subprocess:'at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:208)'
DEBUG:subprocess:'at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveBlock.parseFrom(Osmformat.java:3521)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.pbf2.v0_6.impl.BlobToBlockMapper.parsePrimitiveBlock(BlobToBlockMapper.java:52)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.pbf2.v0_6.impl.BlobToBlockMapper.apply(BlobToBlockMapper.java:28)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfBlobDecoder.runAndValidate(PbfBlobDecoder.java:360)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfBlobDecoder.run(PbfBlobDecoder.java:373)'
DEBUG:subprocess:'at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)'
DEBUG:subprocess:'at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)'
DEBUG:subprocess:'at java.lang.Thread.run(Unknown Source)'
DEBUG:subprocess:'Exception in thread "pool-6-thread-1" java.lang.OutOfMemoryError: Java heap space'
DEBUG:subprocess:'at com.google.protobuf.IntArrayList.addInt(IntArrayList.java:199)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$DenseInfo.<init>(Osmformat.java:8642)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$DenseInfo.<init>(Osmformat.java:8577)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$DenseInfo$1.parsePartialFrom(Osmformat.java:10299)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$DenseInfo$1.parsePartialFrom(Osmformat.java:10293)'
DEBUG:subprocess:'at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:888)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$DenseNodes.<init>(Osmformat.java:12651)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$DenseNodes.<init>(Osmformat.java:12578)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$DenseNodes$1.parsePartialFrom(Osmformat.java:14081)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$DenseNodes$1.parsePartialFrom(Osmformat.java:14075)'
DEBUG:subprocess:'at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:888)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveGroup.<init>(Osmformat.java:4635)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveGroup.<init>(Osmformat.java:4574)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveGroup$1.parsePartialFrom(Osmformat.java:6604)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveGroup$1.parsePartialFrom(Osmformat.java:6598)'
DEBUG:subprocess:'at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:888)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveBlock.<init>(Osmformat.java:3113)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveBlock.<init>(Osmformat.java:3048)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveBlock$1.parsePartialFrom(Osmformat.java:4432)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveBlock$1.parsePartialFrom(Osmformat.java:4426)'
DEBUG:subprocess:'at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:158)'
DEBUG:subprocess:'at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:191)'
DEBUG:subprocess:'at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:203)'
DEBUG:subprocess:'at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:208)'
DEBUG:subprocess:'at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.osmbinary.Osmformat$PrimitiveBlock.parseFrom(Osmformat.java:3521)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.pbf2.v0_6.impl.BlobToBlockMapper.parsePrimitiveBlock(BlobToBlockMapper.java:52)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.pbf2.v0_6.impl.BlobToBlockMapper.apply(BlobToBlockMapper.java:28)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfBlobDecoder.runAndValidate(PbfBlobDecoder.java:360)'
DEBUG:subprocess:'at org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfBlobDecoder.run(PbfBlobDecoder.java:373)'
DEBUG:subprocess:'at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)'
DEBUG:subprocess:'at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)'
DEBUG:subprocess:'Exception in thread "pool-4-thread-1" java.lang.OutOfMemoryError: Java heap space'
DEBUG:subprocess:'Exception in thread "pool-5-thread-2" java.lang.OutOfMemoryError: Java heap space'
DEBUG:subprocess:'Exception in thread "pool-6-thread-2" java.lang.OutOfMemoryError: Java heap space'

FrenzyNatra avatar Oct 21 '22 12:10 FrenzyNatra

Thanks for checking again.

Normally the downloaded and temporary files are not a problem and if there are issues one can go ahead with -fp and -fd. I think we kind of messed up with your files (the used versions) and that's why the removal brought success here.

Finally, we nailed it down to this Java heap error. And: Congrat's on setting this up in VSCode! I know it's kind of a struggle, thats why I published the project via PyPI.

I made an adjustment on the to-be-used memory and published 2.1.0a18, you can also checkout branch issue-156-2 in VSCode as you like. Shouldn't behave different in this case.

Thanks!

treee111 avatar Oct 25 '22 20:10 treee111

Hi @FrenzyNatra did you find time to try it with the solution provided?

treee111 avatar Nov 11 '22 16:11 treee111

Hey :) Already performed some runs, but couldn't finish yet due to some urgent other tasks this week. Short answer: it seems that the map creation finishes successfully when I manage the memory usage to stay somewhere below 70%. Get back tomorrow with more details...

FrenzyNatra avatar Nov 12 '22 17:11 FrenzyNatra

Yayy, it worked 🙌 tested with -nbc parameter set, but as mentioned, memory usage shouldn't completely be disregard, so as long as usage stays below 70% map creation runs through fine 👍 In the first place I had some issues with Java (see below), but installing the latest JDK and a prophylactic reinstallation of wahoomc (2.1.0a18) solved the problem. Many thanks for fixing this 💪 if you need getting tested something special, just tell me. Next I'll try repeating this incl. border countries on 3.0.0.

INFO:# Merge splitted tiles with land an sea INFO:+ Merging tiles for tile 1 of 57 for Coordinates: 137,88 INFO:# Sorting land* osm files DEBUG:subprocess:"Error: missing server' JVM at C:\Program Files (x86)\Java\jre1.8.0_341\bin\server\jvm.dll'." DEBUG:subprocess:'Please install or use the JRE or JDK that contains these missing components.' ERROR:Error in Osmosis with sorting land* osm files of tile: 137,88

FrenzyNatra avatar Nov 13 '22 21:11 FrenzyNatra

Nice! 👍 Thanks for checking again! I didn't want to push you, was just curious if everything is working or something still struggling. I published 3.1.1a0 for you or you can use branch bugfix-java-heap-error. This one has only the productive change added on top of develop and not the load of logging. Would be nice if you could check this combination and then it might be ready to be published!

PS: You can now verbose output via -v. I think you were the one who requested it ;-)

treee111 avatar Nov 13 '22 23:11 treee111

Hey @treee111, yeah sure! All fine, just didn't want to let you waiting so long to close the issue 😄 Checked the version you suggested, this time even on 80 % memory (base load), and all went through smoothly 💪 🥳

Thanks for that great job!!

Version: 3.1.1a0 Args: python -m wahoomc cli -co germany -nbc -fp -md 60 Output:

[...]
INFO:# Create: .map.lzma files
INFO:+ Country: germany
INFO:+ Copying .map.lzma tiles to output folders
INFO:+ Create .map.lzma files: OK

FrenzyNatra avatar Nov 21 '22 20:11 FrenzyNatra

Hi @FrenzyNatra, nice, that is works with the new version as well (I assumed it, better double check). Also with 80% or RAM in usage. 🔢 Gonna merge this now and this issue will be closed then.

Thanks again for posting this and staying close with testing here! 👍

treee111 avatar Nov 23 '22 16:11 treee111