Very long delay between end of simulation and end of execution (for big scenes)

[han16nah]

There is an increasing delay between the end of the simulation and the end of execution with increasing scene size/computational burden of the simulation. For my 8.1 GB .scene file, there are 65 minutes between writing the last point cloud file and end of execution. The simulation itself took only around 23 minutes.

The long delay happens after the last simulation messages are printed:

Waypoint reached!
stepCount = 72520359
Main thread simulation loop finished in 480 sec.
Waiting for completion of pulse computation tasks...
Pulse computation tasks finished in 480 sec.
Total simulation time: 0:08:0

Further info:

• LAS files and trajectory files are written
• Most LAS files are between 500,000 and 625,000 points
• Total number of points for all LAS files: 11,221,123 points
• However, all LAS files are already written when the delay happens.
• The application still heavily uses RAM during this delay

[miguelyermo-heidelberg]

@han16nah, could you provide me the surveys and models to reproduce your results? Thanks in advance.

[han16nah]

Hi @miguelyermo-heidelberg please (temporarily) find the surveys and scene file here: https://heibox.uni-heidelberg.de/d/21d68f719a5041a5825d/ If starting the survey is not working, just tell me and I will upload the individual sceneparts. Best Hannah

[miguelyermo-heidelberg]

Hi @han16nah, it seems that Helios needs the xml scene files to load an already built .scene file. I am sorry to bother you with this. Could you upload them for me? Thanks in advance.

(Maybe is a good idea to remove this restriction by allowing Helios to run with just the survey file + the .scene file, what do you think?)

[han16nah]

Hi Miguel,

find the XML file attached. I hope it is the right one. Let me know if you can now run it if you place the XML file into the same folder as the .scene file. B_BR05_1.zip

[miguelyermo-heidelberg]

Now Helios is finding the .scene file, but throws an exception when reading it. It seems that is because the scene was built on a different machine / architecture than mine. In order to build the scene on my own, I would need all the .obj files, specifically all the contents you have in the "data/sceneparts/SYSSIFOSS/B_BR05_1/" directory.

[han16nah]

Okay, I was afraid of that. https://heibox.uni-heidelberg.de/f/b1155ab7483147369ff6/ I hope you find everything you need in here. I put an additional scene file in there which creates a smaller scene (5 cm voxel size instead of 2 cm) in case the other one one can not be built on your machine.

[miguelyermo-heidelberg]

It seems that 16GB of RAM are not enough to build any of those scenes. I will try as soon as I can in a remote machine with 32GB, If that's not enough, well, I'll figure out another solution. Thank you very much for your attention.

For the sake of curiosity, what type of scene is this? A forest or something like that?

[han16nah]

Yes, it's a forestr scene. Can you try the survey below? It should work with the sceneparts I sent earlier and will load faster than the others. And (although less significant) I also noticed the delay at the end of that survey. als_scenario_scaled_cog_B_BR05_1_wS1_5.zip Sorry for the hassle ;)

Best Hannah

[miguelyermo-heidelberg]

I think I am missing something here. I just tried that scaled survey, which uses the same scene that I can't build on this machine. "data/scenes/Paper/scaled_cog/B_BR05_1.xml#B_BR05_1" is what the survey tries to load, is that correct?

[han16nah]

Yes, that is correct, but I forgot to provide the scene file. I added it in the heibox link: https://heibox.uni-heidelberg.de/f/b1155ab7483147369ff6/ And will also upload it here: B_BR05_1.zip (has to be placed into data/scenes/Paper/scaled_cog)

[miguelyermo-heidelberg]

@han16nah This bug is driving me crazy. Ill took some fresh air working on another topic for a few days. Some questions: What amount of RAM does your system use when building the whole scene? It seems that 32GB are not enough yet, even for the scaled version. Are you executing the simulation by means of PyHelios? After Total simulation time: 0:08:0 is printed, how do you know that helios is still doing background tasks? As I could observe, that print is the very last line of the executed code.

I have a vague idea of what could be happening, but I need you to answer these questions to be sure.

Thanks in advance.

[han16nah]

I'm using a PC with 64 GB RAM.. hm, I would've expected that at least the scaled version works. Are you sure it is because of RAM or did I maybe miss providing some files?

No, I'm executing via the command line. When I execute multiple surveys in a row (i.e. pasting several lines to the command line), it does not immediately start the next task, but "pauses" (for said 65 minutes for the really large scene) while showing this last line Total simulation time: ... of the current simulation.

[miguelyermo-heidelberg]

Yes, I am sure it's a RAM capacity problem on my side. Could you provide me an example of the command you use to perform multiple simulations in a row?

[han16nah]

Simply like this:

run\helios data\surveys\Paper\als_scenario_scaled_B_BR05_1.xml --lasOutput
run\helios data\surveys\Paper\als_scenario_scaled_B_BR05_2.xml --lasOutput
run\helios data\surveys\Paper\als_scenario_scaled_F_BR02_1.xml --lasOutput
run\helios data\surveys\Paper\als_scenario_scaled_F_BR02_2.xml --lasOutput
run\helios data\surveys\Paper\als_scenario_scaled_F_BR08.xml --lasOutput

[FelixEickeler]

Is this still a thing? I am running some automatic generated, very big scenes (>12 GB) and sometimes it seems helios get's stuck after finishing the simulation e.g. after printing the "Total simulation time". Helios keeps sitting on the RAM but there is no CPU time allocated anymore. It feels happening more often if I hit it with more cores. I can't reproduce this though, it is happening on different simulations "randomly".