Suggestion regarding computer specification

[mashrur93]

Hi,

I am trying to run MATSIM for a network with 262144 nodes, 1048576 links,s and 10620 transit routes. The network size is around 7100 square km. The initial population file has 214336 agents with 613861 trips in total. The final goal is to simulate 4.8 M agents with 13.2 M trips.

I ran it on a pc with intel core i7 and 24 GB ram. But I am getting stuck after simulating a certain number of people. The last line of the log file. Then it runs for days, with no output. Also, I ran the code with a pc having 64 GB ram, allocating 56 GB dedicated to matsim. But still, it is using a max of 30 GB ram and continues to run without ending or giving any output.

Any help in this regard would be highly appreciated.

Thanks

logfile.log

[steffenaxer]

The required memory depends heavily on the number of plans per person. Regular this are 5 plans. Moreover the lvl of network resolution will increase the routes, stored in the plan. Merging small consecutive links might reduce the memory footprint. From my experience I can state, that Matsim runs with 1.4 M agents (5.8 M trips) without problems, using 128 gb. But I have never exactly tested the required memory.

[balacmi]

Maybe I can add just a bit more to the discussion. I see that your simulation is hanging during the PersonPrepareForSim. This is probably due to memory requirements for routing of the car/pt trips in your initial demand.

Which MATSim version are you using? v12.0 had some issues with memory for pt routing. Otherwise, as @steffenaxer suggested, you should merge some of the links in your network to reduce the memory needs. Maybe also using SpeedyAlt as a car router could help if you are using the latest MATSim version.

[mashrur93]

Thanks, @steffenaxer, and @balacmi for the suggestions. I am using the latest version 13.

As I mentioned earlier, the network has nearly 1M links, I tried to merge some of the shorter links, but the number is huge. Do you have any suggestions to merge links? I would highly appreciate it. Just to let you know I generated the network from osm and corrected it where necessary (like a link in the opposite direction than it should be directed).

I also rerun the model in a pc with 64 Gb and assigned 57GB before running the model. But I saw the used ram did not exceed 30 GB. (kindly see the attached image for reference). How can I assure that the whole allocated ram is used?

Thanks in advance. matsim_run_error.zip

[steffenaxer]

Maybe some more hints. I can remember a very similar case (see https://github.com/matsim-org/matsim-libs/issues/1277 ). In this situation @mrieser assumed that the plan router is caught in an endless loop. PersonPrepareForSim calls also the PlanRouter, so it might be the very same problem here. Please ensure that you have no links with zero or negative length. Moreover, have a look on the NetworkCleaner class in MATSim to ensure no dead ends in your network. Finally, try out the NetworkSimplifier, it allows you to merge links systematically. Last resort, strip down your network and population to smaller chunks and investigate, when the problem occurs. My personal suggestion: As there is no out of memory exception, I think you have links with zero length, that create routing loops.

[Janekdererste]

1M links is not too much and shouldn't cause problems when assigning the amount of RAM you have. You should make sure that your network and your population are set up correclty.

The current suggested way of creating networks from OSM can be found here: https://github.com/matsim-org/matsim-code-examples/blob/13.x/src/main/java/org/matsim/codeexamples/network/RunCreateNetworkFromOSM.java

This example also includes the usage of NetworkCleaner, although the router should crash in case no route can be found due to an unconnected network. Maybe this helps

[mashrur93]

thanks a lot, @steffenaxer and @Janekdererste. I am working on it now. will update here if the issue is resolved.

[mashrur93]

Hi Again! I solved the issue by analyzing all the agents. While doing that, I faced the following issues.

Regarding Network: a. I tried using networkCleaner, but don't know why after running that the swissRailraptor was constantly giving errors (attached NCerr_logfile.txt). after going through the networkCleaner code, it seems like it completely removes links that cannot be reached by all the links. But, I found some cases, due to missing links somewhere in the midsection of the road(connected links), while creating a network file from osm, the whole road section is removed (attached NCerr.png). Is it possible to add a check if the consecutive links have two-ways or not before completely removing it. However, removing all links with zero length and adding missing links has solved my current issue.

b. While adding link to activity to a link, for some agents it was observed to be adding a link with "pt" network mode (prob_plan.xml), for which no car route was found, leading to a never-ending loop. Another issue was that, as I randomly assigned coordinates to the activity, some coordinates fell into a large space with no nearest link to assign to the activity. Is there any way to add restrictions for adding links to activities such as assigning links with specific modes, maximum distance threshold to look for candidate link?

Strategy: The model falls into a never-ending loop if I add reroute in the strategy. what might be the possible reason for that. Other strategies worked fine though. but the research interest is to study the reroute strategy. Is there a way to address that?

Memory usage error: As I mentioned earlier, a desktop with 64 GB ram was used to run the matsim model. 58 GB was assigned. As you see in the logfile(itr_logfil.txt), the maximum size was used is only 29GB, around 50% of the allocated memory. Is it normal? How can I maximize the memory usage to faster the iterations?

I will highly appreciate it if any suggestions regarding the above-mentioned issues are provided. Thanks in advance.

MATSIM_issues.zip

[Janekdererste]

I tried using networkCleaner, but don't know why after running that the swissRailraptor was constantly giving errors

Make sure to run the network cleaner before you merge your pt schedule into the network. Otherwise the pt links are cleaned away I would assume. If you have multiple network modes, you can use the MultimodalNetworkCleaner like here for example.

The error from NCerr_logfile.txt could be related to that. The message is not very helpful but it might be, that the router is searching for a node which was cleaned away. You have to make sure that your transit schedule and your network correspond to each other, since the schedule has references to the network.

Is there any way to add restrictions for adding links to activities such as assigning links with specific modes, maximum distance threshold to look for candidate link

I guess it depends on what utility you use for that job. If you want to assign links to activities your self, there is a NetworkUtils.getNearestLink(network, coordinate) method. You can then apply your constraints yourself.

How can I maximize the memory usage to faster the iterations?

If you have more RAM assigned than you ever use, you are good. There is no performance benefit in having more RAM. The only speed improvement you get from more RAM is less garbage collection cycles. If this is not a constraint like in your case, the next best thing to speed things up is to buy a faster CPU.