Error while creating partition for the world dataset

[mguzmann]

I am trying to build the whole osm planet dataset. The first step worked fine with:

> osrm-extract -p /opt/foot.lua ./data/world/planet-latest.osm.pbf

But creating the partition failed with:

> osrm-partition data/world/planet-latest.osrm
[info] Computing recursive bisection
[info] Loaded compressed node based graph: 768643066 edges, 1898609519 nodes
[info]  running partition: 128 1.2 0.25 10 1000 # max_cell_size balance boundary cuts small_component_size
[info] Found 1607330894 SCC (849 large, 1607330045 small)
[info] SCC run took: 276.501s
[info] Full bisection done in 18813.7s
[info] Loaded node based graph to edge based graph mapping
[info] Loaded edge based graph for mapping partition ids: 3127067446 edges, 768641854 nodes
[info] Fixed 40068 unconnected nodes
[info] Edge-based-graph annotation:
[info]   level 1 #cells 2556590 bit size 22
[info]   level 2 #cells 270044 bit size 19
[info]   level 3 #cells 17073 bit size 15
[info]   level 4 #cells 562 bit size 10
[info] Renumbered data in 12067.1 seconds
terminate called after throwing an instance of 'osrm::util::exception'
  what():  Can't pack the partition information at level 4 into a 64bit integer. Would require 66 bits.
Aborted (core dumped)

What could be the issue? Is it lack of memory? the server has over 600 gb of free ram.

[danpat]

@mguzmann OSRM can't partition the entire planet with the MLD algorithm as it stands - there are too many edges for the fixed data structure sizes we have allocated.

You have a couple of options:

1. Easiest: Switch to using the CH algorithm (osrm-contract)
2. Moderate: Cut up your OSM file before processing, perhaps into separate continents, and generate two datasets with MLD.
3. Hardest: Make a PR that adds support for >64 bit partition level information.

[mguzmann]

Thanks for your answer @danpat . I tried

> osrm-contract data/world/planet-latest.osrm

But I get:

[info] Input file: data/world/planet-latest.osrm
[info] Threads: 192
[info] Reading node weights.
[info] Done reading node weights.
[info] Loading edge-expanded graph representation
[info] merged 3682928 edges out of 3130750374
[info] initializing node priorities... ok.
[info] preprocessing 768641854 (100%) nodes...
[info] . 10% . 20% . 30% .Segmentation fault (core dumped)

Is it me or my server?

[mjjbell]

For osrm-contract you're probably hitting the same problem as #6169

For osrm-partition there is another option, which is to increase the cell sizes used in creating the partition, thereby reducing the number of cells that need to be packed into 64 bits - see https://github.com/Project-OSRM/osrm-backend/issues/6121#issuecomment-916130387

The downside to this is you're increasing the search space at each level, so you'd expect it to have some effect on query performance, although it's probably negligible relative to other costs incurred during an API call.