Annotations give nodes that don't exist in the input osm file

[flacombe]

Dear maintainers,

As you may find in the attached .osm file, all nodes have really big ids > 9999999999. loire.osm.zip

When I process it with osrm-contract and make it available with osrm-routed 5.19.0, a query like

https://.../route/v1/driving/4.484138488769532,45.541510228130996;4.4870996475219735,45.54508696616638?overview=false&alternatives=true&steps=false&annotations=nodes

gives me the following annotations

{"code":"Ok","waypoints":[{"hint":"sQAAgNwAAIDlAQAAHQAAAFICAAAVBgAADxBCQjBdOEBL_m1C_bIqQzlEAAANBAAAilMAAB3bAAABAAAAhWpEACHptgIqbEQAhui2AgIAvwEuiaZu","location":[4.483717,45.541665],"name":"126623790"},{"hint":"SQEAgGcBAIA4AQAAmQAAAAAAAADuAAAAZvL5QSLDdEEAAAAA3yk6QtkrAAB6FQAAAAAAAHshAAABAAAArHdEAJH2tgK8d0QAf_a2AgAArwMuiaZu","location":[4.487084,45.545105],"name":"349379862"}],"routes":[{"legs":[{"steps":[],"weight":537.4,"distance":537.4,"annotation":{"nodes":[1604625344,2455888032,6010937664,7256645296,6208998656,4508343296,3635029600,939062000,6278481792,3977370144,5618166736,1851168448,3461536656,5512249904,7342764128,4935365664,1502573568,2345701904,7905357696,1214328448,4647535552,7875718336,4701186976,721583008,7450880000,7671176016,3106836464,7839090624]},"summary":"","duration":19344.7}],"weight_name":"surface","weight":537.4,"distance":537.4,"duration":19344.7}]}

Any of returned node don't exists in the input file. Why?

What should I do to get a consistent answer with existing nodes please?

[danpat]

Looks like we've got an overflow bug storing node IDs.

In your file, one of the node ID values is 99742940620696000 - this is waay bigger than 2^32, but is still less than 2^64. I suspect we've got a signed-vs-unsigned integer problem in here somewhere.

The quick fix on your side would be to make sure node ID values are less than 2^32 (4294967296), which should avoid triggering the overflow.

[flacombe]

Thank you @danpat for quick answer

The quick fix on your side would be to make sure node ID values are less than 2^32 (4294967296), which should avoid triggering the overflow.

Unfortunately I can't easily. My ids are kind of hashcode intended to deduplicates nodes according to their coordinates while producing the osm xml Ids format depends on the grid I use to snap coordinates (6 decimal of epsg:4326 on the I area I'm working on) The only chance I have to lower the ids size is to renumber nodes once deduplicated in a table containing near of 100M ways/nodes tuples. I'd better don't touch it :)

[danpat]

Ok, here's where the problem is in the code:

https://github.com/Project-OSRM/osrm-backend/blob/82b5648c97edf1d2edec7aecebc35aa8a8033c82/include/extractor/packed_osm_ids.hpp#L14

We're packing OSM IDs into a 33 bit value to save some space, as "real" OSM IDs haven't grown that big yet. You're using values that require 57 bits of storage space (at least in your example), so lots bits are being discarded.

You will need to recompile OSRM and change the line above from 33 to 57 or higher (max of 64). You will need to regenerate your datafiles after this change, as it will make the code incompatible with the datafiles you currently have.

[flacombe]

This is precisely what I needed to read tonight Thank you very much.

[flacombe]

So I've compiled 5.21 with vector length 60 instead of 33. Thank you @danpat

Initial problem sounds to be solved as annotations ids now have the proper length.

Nevertheless, with the same osm file as provided in the first post, there are still problems with nodes order

Let's take the following query : route/v1/driving/4.486756324768067,45.542652404503066;4.487485885620118,45.54558288246844?overview=false&alternatives=true&steps=false&annotations=nodes

which gives the following output :

"annotation":{"nodes":[99997567507608000,99999048455007008,100001185822402000,100003219123306000,100007349766126000,100007157643246000,100005404521968992,100004179738536992,100000609454712992,99999905004120992,99998936384556992,99997279324640000,99999536768920000,100002058382180000,100000801578304992,99999592805176992,99998648200936000,99997583520000000,99997815668530000,99998936385622000,99997775643528000,100003147079794000,100005044293543008,100006197031056992]}

Two first node #99997567507608000 #99999048455007008 are found in the input file, but aren't consecutive. Am I misunderstanding the annotations output as to take it as a single string of consecutive nodes?

#99999048455007008 is in the middle of a way, the only possibility of preceding node is #100001185822402000 according to me

Any mistake in my logic?

EDIT : Big mistake was to not look on the opposite direction on the way :1st_place_medal: Nodes #99997567507608000 #99999048455007008 are actually consecutive. Problem solved :)

[danpat]

Hah, I was just reading my email and I found the same thing - glad you uncovered it on your own.

I think this ticket should stay open until we fix the PackedOSMIDs feature - at a minimum, we should emit a warning or error when someone tries to pack a value bigger than we have space for. I don't like silently doing the wrong thing like we're doing currently.

[flacombe]

Ok for me

[stevenae]

Just encountered this myself.