Receiving "No such file or directory: /data/us-latest.osrm.fileIndex" when generating osrm-partition

[gbranne1]

Issue

I am processing large OpenStreetMap (OSM) extracts using OSRM on a high-memory EC2 instance provisioned via AWS CDK. I have plenty of memory on the AWS EC2. The workflow involves downloading .osm.pbf files, running the OSRM preprocessing pipeline (osrm-extract, osrm-partition, osrm-customize, osrm-routed), and storing the results in S3 bucket. The main issues encountered is the inability to generate the us-latest.osrm.fileIndex when performing the following command: docker run -t -v "${PWD}:/data" ghcr.io/project-osrm/osrm-backend osrm-partition /data/us-latest.osrm I am getting an error that seems to unable open the file : data/us-latest.osrm.fileIndex. Also something mentioning about no finding the mmap_file.hpp

Steps to reproduce

Please provide the steps required to reproduce your problem.

1. I was successful in downloading the us-latest.osm.pbf.

2. I successfully ran : docker run -t -v "${PWD}:/data" ghcr.io/project-osrm/osrm-backend osrm-extract -p /data/car.lua /data/us-latest.osm.pbf

3. It created the following files: -rw-r--r-- 1 ssm-user ssm-user 14993 May 17 23:13 car.lua -rw-r--r-- 1 ssm-user ssm-user 11120702889 May 9 20:42 us-latest.osm.pbf -rw-r--r-- 1 ssm-user ssm-user 1197477888 May 17 23:13 us-latest.osrm.cnbg -rw-r--r-- 1 ssm-user ssm-user 1198218240 May 19 04:31 us-latest.osrm.cnbg_to_ebg -rw-r--r-- 1 ssm-user ssm-user 6587589632 May 17 23:14 us-latest.osrm.ebg -rw-r--r-- 1 ssm-user ssm-user 1746078720 May 17 23:14 us-latest.osrm.ebg_nodes -rw-r--r-- 1 ssm-user ssm-user 1921390592 May 17 23:14 us-latest.osrm.edges -rw-r--r-- 1 ssm-user ssm-user 1723622912 May 17 23:14 us-latest.osrm.enw -rw-r--r-- 1 ssm-user ssm-user 10367034880 May 17 23:14 us-latest.osrm.geometry -rw-r--r-- 1 ssm-user ssm-user 1125590016 May 17 23:15 us-latest.osrm.icd -rw-r--r-- 1 ssm-user ssm-user 7680 May 17 23:15 us-latest.osrm.maneuver_overrides -rw-r--r-- 1 ssm-user ssm-user 50269184 May 17 23:15 us-latest.osrm.names -rw-r--r-- 1 ssm-user ssm-user 3444668928 May 17 23:15 us-latest.osrm.nbg_nodes -rw-r--r-- 1 ssm-user ssm-user 6144 May 17 23:15 us-latest.osrm.properties -rw-r--r-- 1 ssm-user ssm-user 23602688 May 17 23:15 us-latest.osrm.ramIndex -rw-r--r-- 1 ssm-user ssm-user 4096 May 17 23:15 us-latest.osrm.restrictions -rw-r--r-- 1 ssm-user ssm-user 3584 May 17 23:15 us-latest.osrm.timestamp -rw-r--r-- 1 ssm-user ssm-user 20480 May 17 23:15 us-latest.osrm.tld -rw-r--r-- 1 ssm-user ssm-user 39424 May 17 23:15 us-latest.osrm.tls -rw-r--r-- 1 ssm-user ssm-user 548968960 May 17 23:15 us-latest.osrm.turn_duration_penalties -rw-r--r-- 1 ssm-user ssm-user 3293794816 May 17 23:15 us-latest.osrm.turn_penalties_index -rw-r--r-- 1 ssm-user ssm-user 548968960 May 17 23:16 us-latest.osrm.turn_weight_penalties

4. I tried to run the following command: docker run -t -v "${PWD}:/data" ghcr.io/project-osrm/osrm-backend osrm-partition /data/us-latest.osrm

5. I get the following error: terminate called after throwing an instance of 'osrm::util::exception' what(): File "/data/us-latest.osrm.fileIndex" mapping failed: failed opening file: No such file or directory: iostream errorinclude/util/mmap_file.hpp:33

OSRM Information:

• osrm-backend version being used : h-4.2$ docker run ghcr.io/project-osrm/osrm-backend osrm-extract --version v6.0.0

• OSM extract that was processed docker run -t -v "${PWD}:/data" ghcr.io/project-osrm/osrm-backend osrm-extract -p /data/car.lua /data/us-latest.osm.pbf

• Processing commands (e.g. CH vs MLD processing) Here is the sequence I was trying to do. osrm-extract worked but osrm-partition did not work: # 1. Extract docker run -t -v "${PWD}:/data" ghcr.io/project-osrm/osrm-backend osrm-extract -p /data/car.lua /data/us- latest.osm.pbf

  2. Partition

   docker run -t -v "${PWD}:/data" ghcr.io/project-osrm/osrm-backend osrm-partition /data/us-latest.osrm
  

  3. Customize

   docker run -t -v "${PWD}:/data" ghcr.io/project-osrm/osrm-backend osrm-customize /data/us-latest.osrm
  

  4. Serve

  docker run -t -i -p 5000:5000 -v "${PWD}:/data" ghcr.io/project-osrm/osrm-backend osrm-routed --algorithm mld /data/us-latest.osrm
  

• Server queries

If you're reporting an issue with https://map.project-osrm.org, please provide a link to the problematic request.

System Specifications

Please provide details of your development environment.

• Library/dependency versions
• Operating system
• Hardware Operating System Amazon Linux 2 (EC2 instance, 256 GB RAM, 200 GB EBS) Also tested on Windows 11 (for S3 upload and orchestration)

Hardware AWS EC2 Instance Type: r5.16xlarge (256 GB RAM, 64 vCPUs) Storage: 200 GB EBS volume (gp3) Network: VPC with SSM and S3 endpoints

Key Libraries / Dependencies OSRM Docker Image: ghcr.io/project-osrm/osrm-backend:latest AWS CLI: Version 2.x Docker: Version 20.10.x (installed via amazon-linux-extras) AWS CDK: Java CDK v2.x (see software.amazon.awscdk.* imports) Java: OpenJDK 11 (for CDK stack) Python: 3.9 (for Lambda functions) Node.js: v18.x (for CDK CLI, if used)