Error `Required files are missing, cannot continue` with local Docker

[nikkatalnikov]

trafficstars

docker run -d -p 5000:5000 -e OSRM_PBF_URL='http://download.geofabrik.de/europe/ukraine-latest.osm.pbf' --name osrm-backend peterevans/osrm-backend-k8s:latest

docker logs -f 301094bac9b726ccf7c83ab4567a625903a6f5794e8903185c90fb220bfa4e1d

produces the following:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  4  417M    4 17.2M    0     0  26723      0  4:33:10  0:11:18  4:21:52     0
curl: (18) transfer closed with 419883119 bytes remaining to read
[info] Parsed 0 location-dependent features with 0 GeoJSON polygons
[info] Using script /osrm-profiles/car.lua
[info] Input file: data.osm.pbf
[info] Profile: car.lua
[info] Threads: 2
[info] Parsing in progress..
[info] input file generated by osmium/1.8.0
[info] timestamp: 2018-05-28T20:00:02Z
[info] Using profile api version 4
[info] Found 3 turn restriction tags:
[info]   motorcar
[info]   motor_vehicle
[info]   vehicle
[info] Parse relations ...
TBB Warning: Exact exception propagation is requested by application but the linked library is built without support for it
terminate called after throwing an instance of 'tbb::captured_exception'
  what():  PBF error: truncated data (EOF encountered)
/docker-entrypoint.sh: line 61:     7 Aborted                 osrm-extract $OSRM_DATA_PATH/$OSRM_DATA_LABEL.osm.pbf -p /osrm-profiles/$OSRM_GRAPH_PROFILE.lua
[warn] Missing/Broken File: /osrm-data/data.osrm.ebg
[warn] Missing/Broken File: /osrm-data/data.osrm.ebg_nodes
[warn] Missing/Broken File: /osrm-data/data.osrm.properties
[warn] Missing/Broken File: /osrm-data/data.osrm.ramIndex
[warn] Missing/Broken File: /osrm-data/data.osrm.fileIndex
[warn] Missing/Broken File: /osrm-data/data.osrm.edges
[warn] Missing/Broken File: /osrm-data/data.osrm.geometry
[warn] Missing/Broken File: /osrm-data/data.osrm.turn_weight_penalties
[warn] Missing/Broken File: /osrm-data/data.osrm.turn_duration_penalties
[warn] Missing/Broken File: /osrm-data/data.osrm.datasource_names
[warn] Missing/Broken File: /osrm-data/data.osrm.names
[warn] Missing/Broken File: /osrm-data/data.osrm.properties
[warn] Missing/Broken File: /osrm-data/data.osrm.icd
[warn] Missing/Broken File: /osrm-data/data.osrm.maneuver_overrides
[error] Required files are missing, cannot continue

[nikkatalnikov]

peterevans/osrm-backend-k8s:1.16 works well. seems like osrm-backend bug

[peter-evans]

The problem is on the fourth line of your log file. The pbf file failed to download properly and was incomplete.

curl: (18) transfer closed with 419883119 bytes remaining to read

The script could handle that better I suppose and maybe retry the download a few times before failing. Checking against the md5 sum of the file would also be a good feature to add.