Multipolygon missing

[yvecai]

Relation 1622434 missing from database.

Context

My database is running updates every 10 minutes since 2019-12-05T00:52:02Z with this config opensnowmap.yml. A user pointed to me a few forest multi-polygons missing from the rendering.

Expected Behavior

imposm=# select count(*) from osm_landusages where osm_id = -1622434; should return '1', I get '0'

Possible Fix

Tested locally with imposm-0.11.1-linux-x86-64: Re-importing the actual relation from an Overpass API extract makes it in the database without trouble. Re-importing the old version of the relation and references extracted with osmium from the old planet file makes it in the database without trouble. JOSM shows no complaint on this old relation extract.

Your Environment

• Imposm build from just after this commit https://github.com/omniscale/imposm3/commit/c3c03058312e68e482c30cbfa6191eff25bc179b so, with GEOS 3.7.3
• postgresql-11-postgis-2.5
• Debian Buster

[yvecai]

In addition,the relation is present in cache: imposm query-cache -cachedir "~/imposm_cache" -rel 1622434 { "relations": { "1622434": { "ID": 1622434, "Tags": { "landuse": "forest", "type": "multipolygon" }, "Metadata": null, "Members": [ { "ID": 255814912, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 117446045, "Type": 1, "Role": "outer", "Way": null, "Node": null, "Element": null }, { "ID": 24535783, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 24537888, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 24535782, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 24537969, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 24538179, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 24537891, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 24537892, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 532021568, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 24537890, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 24537889, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 24538081, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 24538082, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 24538170, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 24538253, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 531736878, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 204242795, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 365881645, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 365882774, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 365882773, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 366137049, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 24538252, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 531736869, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 531972504, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 532021569, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null }, { "ID": 636349184, "Type": 1, "Role": "inner", "Way": null, "Node": null, "Element": null } ] } } }

[jordanemitchell]

Same - I'm seeing a few multipolygons with tagging that is matched by the mapping show up in the cache, but not the imported tables.

[thesocialdev]

We are also experiencing something similar, see our ticket bug for more context if you're interested https://phabricator.wikimedia.org/T288897#7288008

[andreasjunghans]

We saw the same effect, and in our case it was actually caused by a bug in the Osmium command line tool: In versions before 1.8.0, the derive-changes command produces node deletions instead of other elements. I.e. if way 123 is deleted, the changes file marks the node with id 123 as deleted.

The problem manifests itself most visibly in multipolygons since they have lots of nodes. Imposm doesn't flag missing polygon nodes in the log, so while the polygons are present in the cache, they are silently skipped in the database.

In case Osmium's derive-changes is the problem, you can add the --keep-details command line flag as a workaround (produces slightly larger files, but doesn't suffer from the node bug), or you can update to a newer version (at least 1.8.0). In other cases, it may simply be a problem in OSM data (see https://github.com/omniscale/imposm3/issues/209).

I think it would be useful if missing nodes were logged by Imposm - however, I don't know if that would flood the log (depending on how common the problem is).

[yvecai]

This is unlikely the issue in my case:

$dpkg -l | grep osmium
ii  osmium-tool                              1.13.1-1~bpo10+1
ii  pyosmium                                 3.1.3-1~bpo10+1
ii  python-pyosmium                          2.15.1-1           
ii  python3-pyosmium                         3.1.3-1~bpo10+1       

[yvecai]

Any ideas are welcome for a workflow detecting those missing MPs: I keep finding them from time to time browsing the map, but I can't find relevant errors in Postgresql logs nor Imposm output.