simulationStep speed doesn't decrease after traffic demand decreases

[nholt01]

We have a simulation which has ~192k persons in it. All of these persons depart at t=0. There are "rush hours" in our traffic demand which cause many vehicles to spawn in and much traffic to be simulated, which drastically slows down SUMO. However, once "rush hour" is over, we expect simulation time to drop as it is now mostly persons stay still ( tags). However, we see that the simulation stagnates, taking around 1.5 hours to simulate 1 hour of traffic during and after "rush hour".

We've been wondering if it might be a case of vehicles not leaving the simulation like they should after reaching their destination, or something similar. The simulation starts off very quick (a few seconds to simulate 1 hour of traffic where most people are in tags) but once rush hour happens it comes to a crawl, and stays there even after traffic on the road network reduces.

Any insight as to why this might be happening? We're using the C++ libsumo library to run the simulation steps. We load in a fully-specified routes file for 24 hours of data, which has ~192k persons in it, all departing at t=0. They all stay stationary (via tags) for the first few hours and then start to move ("rush hour"). Then they stay for another few hours.

Any insight into this, or suggestions of how better to debug would be appreciated. Thank you!

SUMO-version: SUMO 1.18.0

operating system: Ubuntu 22.04 LTS

[behrisch]

It would be great if you could share the scenario or maybe at try to extract a minimal example. Are there subscriptions involved and does the problem disappear if you disable them?

[nholt01]

No subscriptions are involved. We have tested by stripping our code to just calling simulationStep and timing execution time.

Extracting an example might be difficult - the time differences probably aren't noticable with only a couple of persons. We have noticed the slowdown with 50k people - we can take an action to test with lower amounts of people to see, but eventually the simulation is just so quick on low numbers of persons that you probably won't notice the slowdown.

I'd love to hear of any suggestions you might have of how to create a minimal example.

I can describe the basic thing:

<vehicle id="1_1" depart="triggered">
    <route ... />
</vehicle>
<vehicle id="1_2" depart="triggered">
    <route ... />
</vehicle>
...
<person id="1" depart="0">
    <stop until="1000" lane="A" />
    <ride to="B" lines="1_1" />
    <stop until="10000" lane="B" />
    <ride to="C" lines="1_2" />
    ....
</person>

And so on. Each person follows this pattern - alternating between and tags, with each tag having a specified, unique vehicle. The vast majority of a person's daily activity is in tags - each person takes between 3-5 tags during a 24h simulation, and spends >90% of their time stationary (in tags). Repeating this pattern with more persons will show the slowdown in simulation (at least when driven from the C++ libsumo with simulationStep).

Sorry, this is very hard to create a minimal example due to the scale required. If you have any suggestions, I'd be very appreciative!

[namdre]

The --verbose output of the simulation reports time spent for different things can you check how the UPS (updates per second) and UPS-Persons vary when going of the the too-many-persons threshold?

Can you replicate the same thing when running without libsumo? (just starting the sumo binary with a .sumocfg) ?

[behrisch]

I will try to setup a scalability test myself.

[behrisch]

I did some tests with the attached script which basically adds 50000 persons to a network, lets them wait until time 2000 then all of them start driving a car for a few edges and then leave the car and wait until 10000. I see no difference in runtime in the waiting steps before and after driving. (Just for the records: It takes about 0.7ms on my machine to do 500 simulation steps in the waiting regime and 3 to 7 seconds while everyone is driving). @nholt01 Maybe you can use that script as a template and modify it such that it reflects your scenario better. script.zip