ActivitySim multiprocessing reduces the memory usage

[i-am-sijia]

trafficstars

During the ActivitySim Phase 8 2-zone memory profiling task, we observed that accessibility calculation requires large memory. After some tests, we have found that multiprocessing accessibility with large number of processors reduces the peak memory usage. (Users can set numeric num_processors in the settings.yaml for multiprocessing)

Initial tests

Below are the tests done with the full-size placeholder PSRC model, on a 64 core, 512 GB RAM machine:

• Test 1: Run accessibility in single process. Failure, requires more than 512 GB of RAM
• Test 2: Run accessibility with num_processors = 2 (3,4, a small number). Failure, still requires more than 512 GB of RAM
• Test 3: Run accessibility with num_processors = 28. Success, the peak memory is below 512 GB of RAM

Reasons

In theory, multiprocessing does not reduce memory usage, because all processors should be executing simultaneously, and the total memory usage would be similar to that of single processing. However, we noticed not all multiprocessors for accessibility were started and executed simultaneously. Hence instead of parallelism, it was more like chunking and multiprocessors were executed somewhat sequentially. In this case, it had a positive impact on the memory problem.

Unsolved

It is still not clear why ActivitySim's multiprocessors are not executed at the same time. But we have had two hypotheses:

1. One hypothesis is that ActivitySim multiprocessing takes time to create the many HDF5 files that are used by the processors, but this does not seem to be the case after I did some digging.
2. Another hypothesis is that the HDF5 files are slow to read, and each processor hogs the computer’s i/o when reading HDF5, hence there is a wait to execute processors.

There are likely other reasons.

Further tests

To further confirm if multiprocessing can be used as a reliable workaround for the memory issue, I tested an extreme case:

• Test 4: Run accessibility with num_processors = 100 (way more than the number of cores a typical modeling machine has). Success, the peak memory is about 45 GB. It took about 15 minutes to run accessibility.

This confirms that specifying more processors leads to lower memory peak when calculating accessibility. With setting more number of processors, the problem-to-solve is divided into more and smaller pieces/jobs, which means more (and/or excessive) jobs are waiting in the queue to be executed.

Multiprocessing as a short-term workaround to the memory issue

PSRC confirmed they were also able to successfully run accessibility with setting num_processors = 100 on a machine with less RAM and fewer cores.

Moving forward, as a short-term workaround, users can specify a large MP number (like 100) for 2-zone accessibility calculation to significantly reduce the memory requirement. It could also be a solution to other memory-intensive sub-models. ActivitySim MP allows varying num_processes by sub-model.