osm2pgrouting Strange stray lines in import

Imported this way:

kom@nucat:~$ osmconvert BY.osm.pbf --out-osm --drop-author --drop-version  > by.osm
kom@nucat:~$ osm2pgrouting -f by.osm -d kom -W 111 -U kom

Dump copy: https://www.dropbox.com/s/w5q39je7wps6sk7/BY.osm.pbf?dl=0

osm2pgsql does not create these stray lines.

Aug 21 '18 17:08 Komzpa

Hi @Komzpa . What version are you using?

Aug 24 '18 17:08 cayetanobv

Hi, it was latest shipped with Ubuntu Bionic.

Aug 24 '18 17:08 Komzpa

#183 is another instance of this

Aug 27 '18 13:08 Komzpa

Hi @Komzpa ,

Sorry for delay in my response but last week was hard... I've imported Belarus complete dataset from Geofabrik and I don't see these strange lines in output (rendered by QGIS):

A detail of Vitesbk:

Commands used:

$ wget http://download.geofabrik.de/europe/belarus-latest.osm.pbf
$ osmconvert belarus-latest.osm.pbf --drop-author --drop-version --out-osm -o=belarus_reduc.osm
$ osm2pgrouting -c /usr/local/share/osm2pgrouting/mapconfig.xml -f /tmp/belarus_reduc.osm -d osm2pgrouting_tests -h localhost -U postgres -W postgres --schema belarus  --chunk=100000 --clean

Could you try with these data and last osm2pgrouting version?

Thanks,

Sep 04 '18 01:09 cayetanobv

Hi @cayetanobv, This ticket and #183 report the same issue separately. I shared reproducing dump. I don't need the geofabrik dump, I want to use the same dataset across all my tools. OSRM and osm2pgsql handle the dump just fine.

My guess is that osm2pgrouting can't handle nodes absent but referenced in the dump and substitutes it with dirty memory. Can you check it?

Sep 04 '18 01:09 Komzpa

I also remember, that the "Metro Extracts" caused issues, which could be the same problem: https://mapzen.com/documentation/metro-extracts/

Regarding osm2pgsql I assume, that it does not try to "correct" data as osm2pgRouting does. But like @Komzpa says, it could be a data issue with geometries that lie at the border of the extracted area.

Sep 06 '18 05:09 dkastl

@Komzpa

My guess is that osm2pgrouting can't handle nodes absent but referenced in the dump and substitutes it with dirty memory. Can you check it?

Your guess is almost correct.

For routing we need a cost, the cost is based on the latitude and longitude of the geometry, and speed (given in the configuration file). The geometry that osm2pgrouting is the geometry from where the length, cost, reverse_cost, cost_s, reverse_cost_s are calculated from.

here If the node is not on the node section, then it does not have the latitude and longitude, therefore its ignored (we can not calculate cost information needed for routing) so that node is not added.

osm2pgrouting focus is to build a working routing topology (not a geometry topology), based on the information that comes directly from OSM (.osm) with the following structure.

nodes
ways
relations

and within each section the information is ordered by osm_id

Current Simplified Problem:

Get the edge information needed for routing

node data

id	lat	lon
12345	0	1
23456	1	1

Way "1" is a "one way" road and the nodes are 12345, 1, 20, 300, 444, 55, 6666, 23456

Based on the provided information The length of traveling from 12345 to 23456 is 1.4142

The resulting row on the edge table + the geometry of the edge from where it was calculated:

id	source	target	cost	reverse_cost	geom
1	12345	23456	1.4142	-1	LINESTRING(1 0, 1 1)
2	...

What if ...

Resulting edge table is:

id	source	target	cost	reverse_cost	geom
1	12345	23456	1.4142	-1	NULL
2	...

Benefits:

The edge table is ready for routing and you don't see the strange lines in the graph

Prediction:

Open New Issue: osm2pgrouting does not calculate all the edges geometry
You can't visually detect that the graph is not the graph you need for your routing application
Open New Issue: with some routing queries my results are in the middle of nowhere

Another What if ...

Resulting edge table is:

id	source	target	cost	reverse_cost	geom
1	12345	23456	NULL	-1	see "what if" above
2	...

Benefits

NULL because the "complete" cost could not be calculated
Filter the QGIS layer using WHERE cost IS NOT NULL

Prediction:

Open Issue: The edge table is not ready for routing

When importing from osm2pgrouting I get

SELECT * FROM pgr_dijkstra(
    'SELECT gid AS id, source, target, cost FROM ways', 
    1639, 1195, directed := false);
ERROR:  Unexpected Null value in column cost
CONTEXT:  SQL function "pgr_dijkstra" statement 1

Yet Another What if ...

Resulting edge does not contain the offending edge:

id	source	target	cost	reverse_cost	geom
2	...

Benefits

You don't see the weird straight lines

Prediction: Open Issue: I see that way "1" is in the OSM file but the edge table does not have it. Because of my application I processed the osm file removing the nodes that are irrelevant for the cost calculation.

My conclusion

We have plans to rewrite osm2pgrouting to be able to process unordered files, read this but for the moment make sure that the processed file has:

nodes
ways
relations

and within each section the information is ordered by osm_id making sure that all the nodes of the ways that are going to be used for routing are in the file and make a suitable configuration file that filter out the ways that are not going to be used for routing.

If strange straight line remain

if the line is going to be used for routing double check that the all the nodes needed are in the osm file
if the line is not going to be used for routing:
- option one: double check your configuration file
- option two: delete the row from the database

Oct 21 '18 05:10 cvvergara

This ticket and #183 report the same issue separately. I shared reproducing dump. I don't need the geofabrik dump, I want to use the same dataset across all my tools. OSRM and osm2pgsql handle the dump just fine.

@Komzpa , I understand that you don't want to use the Geofabric dump. However, with their extracts I (and probably also others) have never had an issue, but with other OSM extracts we had some.

Vicky tried to understand the matter and what possible solutions exist. However, for us osm2pgrouting at the moment has not as much priority as working on version 3 of pgRouting. And since there is a way to get it work with osm2pgrouting by using Geofabric data, we would be happy to accept any pull request to fix this issue, but for us this does not have that high priority, because there is a high chance that this is a data problem and not a problem of osm2pgrouting.

Would it be better, if osm2pgrouting validates OSM extracts before running the process and refuses to do so, when data is not as expected?

Oct 22 '18 02:10 dkastl

Maybe pgRouting could be run with an optional flag to also take into account the "geometry" of the road links ... just someone hat to implement this -> #GSOC ;-)

Oct 22 '18 02:10 dkastl

Graph problems are not geometry problems pgRouting functions needs: id, source, target, cost, and reverse cost pgRouting functions don't need: geom

ALTER TABLE edge_table DROP COLUMN geom and pgRouting will still work fine.

What could be done is have an extra flag in osm2pgrouting where the user choose: --no-geom don't get the geometries from where the "cost" comes from --no-bad-input don't include the rows where my data has incomplete information --stop-on-bad-input Stop everything when my data has incomplete information

And yes, whatever we decide, do it on next year GSoC program

Oct 22 '18 12:10 cvvergara

Hello,

If nodes are skipped, why do the lines go to seemingly random locations inside the country? Just skipping them as if they never existed will make the image same as osm2pgsql does (which is a good solution).

Oct 25 '18 14:10 Komzpa

It is are not skipping nodes that dont exist on the data. please make sure that for the lines in question, all the node information is there, that you didnt skip them

Oct 25 '18 23:10 cvvergara

Note that the geometries returned are from where the cost is calculated from

Oct 25 '18 23:10 cvvergara

In case this is helpful to anyone looking here for same reason as me, I was running into this issue when when querying the Overpass API with the recursion method shown in most of the examples: <; Digging deeper, I changed it to the "Completed ways and relations" method: <; >; as per https://wiki.openstreetmap.org/wiki/Overpass_API/Advanced_examples#Completed_ways_and_relations and the issue is resolved.

Jul 13 '19 04:07 maciejb

I had the same issue with a file that I produced with osmosis using the version 2.3.3 of osm2pgrouting. With the compiled version 2.2.0 of osm2pgrouting it worked as it should. So maybe this could be useful for somebody. Maybe this is helpful for somebody.