Enhance augmented diffs with version for all objects

[vshcherb]

Some story behind: We are working on OsmAnd live feature to provide incremental map updates based on augmented diffs. I think this could be a very similar task for many rendering engines.

In order to have proper incremental updates, map object "id" and "version" should be identified. OSM version is not enough to keep object versions, the simplest example is to change any "node" location, the "way" version will stay the same in OSM but logically it should be changed cause the polygon/road has changed geometry.

Proposition: Can we please have an option see not only ids (ref) for objects but also their version.

Direction: Can you please help us understand how difficult to implement & we could start in case we can help by creating a pull request.

[mmd-osm]

I'm not sure, if I understood your issue correctly. Even today, you should have all relevant information available in the augmented diff. It is just a matter of putting the different pieces together.

Let's take a look at the following example: http://overpass-turbo.eu/s/pOC (corresponds to: augmented_diff?id=2503120)

Node 322916909 was changed in changeset 49591972. In the augmented diff, this situation is indicated by a modify action along with the old and new version:

<action type="modify">
<old>
  <node id="322916909" lat="52.1453818" lon="0.6862772" version="3" timestamp="2014-11-17T21:56:40Z" changeset="26853522" uid="553735" user="Johnmb"/>
</old>
<new>
  <node id="322916909" lat="52.1453725" lon="0.6862975" version="4" timestamp="2017-06-16T13:35:17Z" changeset="49591972" uid="281267" user="smb1001"/>
</new>
</action>

In addition, as this change also affects way 29357359 (although it hasn't been directly changed), the augmented diff includes an action modify along with an <old> and <new> tag.

This is a clear indication, that the way also needs to be processed due to possible tag/geometry changes. To find out what exactly changed, you need to take previous parts of the augmented diff file into account, most notably, the corresponding <node> id as shown above. By doing this kind of lookup, you will eventually see that node 322916909 now has a new version with changed coordinates.

<action type="modify">
<old>
<way id="29357359" version="10" timestamp="2016-10-14T20:06:34Z" changeset="42905440" uid="76617" user="robbieonsea">
<bounds minlat="52.1407152" minlon="0.6832561" maxlat="52.1537959" maxlon="0.6953002"/>
<nd ref="4448334599" lat="52.1407152" lon="0.6832561"/>
<nd ref="322916918" lat="52.1407762" lon="0.6836332"/>
<nd ref="287892078" lat="52.1410269" lon="0.6837568"/>
...
<nd ref="322916909" lat="52.1453725" lon="0.6862975"/>
...

</way>
</old>
<new>
   <way id="29357359" version="10" timestamp="2016-10-14T20:06:34Z" changeset="42905440" uid="76617" user="robbieonsea">
<bounds minlat="52.1407152" minlon="0.6832561" maxlat="52.1537959" maxlon="0.6953002"/>
<nd ref="4448334599" lat="52.1407152" lon="0.6832561"/>
<nd ref="322916918" lat="52.1407762" lon="0.6836332"/>
<nd ref="287892078" lat="52.1410269" lon="0.6837568"/>
...
<nd ref="322916909" lat="52.1453725" lon="0.6862975"/>
...

[vshcherb]

Your example is not correct http://www.openstreetmap.org/changeset/26853522.

This changeset has way id="29357359", in many editors today when you modify a node it also modifies a way like ID (adding some metainfo to it). Generally it is not true, changing geometry of the way doesn't include way into changeset hence it doesn't show there.

Though I think it doesn't solve the issue in a general way, cause changing geometry doesn't change version of the way itself. That's way proper "version" should be generalized to "a sum of version of all nodes and way version". Then calculating "version" could not be done without all "version" on the nodes.

P.S. Just to repeat in another words, It is not a problem of rendering the change cause there is enough information. It is a problem to generate "a diff map" that could be overlayed on the already existing map and the diff would be integrated into the map without glitches.

P.P.S. Last but not least osmChange contains all versions information, clearly augmented diff outputs much more detail so the format is different, though that information will be particularly valuable as well.

[mmd-osm]

Your example is not correct http://www.openstreetmap.org/changeset/26853522. This changeset has way id="29357359",

The example was ok, my explanation mentioned the wrong cs at one point, it should have read 49591972 instead. I updated my previous answer accordingly. In any case, the changeset needs to be within the analysis time window, that's "2017-06-16T13:35:00Z" till "2017-06-16T13:36:00Z". Augmented diff look at what changed during that time window. Basically, it runs the same query twice, once at "2017-06-16T13:35:00Z" and again at "2017-06-16T13:36:00Z" and compares the differenences of both queries. That's what you see in my example above.

That's way proper "version" should be generalized to "a sum of version of all nodes and way version".

Although I understand your reasoning, I don't think this is the idea behind augmented diffs. The data returned always exactly reflects the data as in the OSM database. Introducing some artificial way version numbers (i.e. version numbers which are not present in the OSM database) would immediately ask for conflicts with future way versions. In addition you would need to keep track of all those node changes, propagate them to way version numbers and keep a full history across the entire way's lifecycle. This is clearly outside of the scope of augmented diffs as they are defined.

Also see http://wiki.openstreetmap.org/wiki/Overpass_API/Augmented_Diffs

Please note that for ways and relations old and new version could have the same version number. This happens when the element itself is unchanged but a member element has changed and thus also the geometry of the element itself.

[vshcherb]

@mmd-osm I think by trying to explain reasoning behind it, the idea is stretching too far where Augmented diffs are exactly not designed.

The request it to display "version" of OSM objects for all objects and not only for specific. Today augmented diff displays only version of changed objects and not inner objects though it prints their ids and coordinates. I don't think it is must have but it could be very much nice to have.

Please comment also whether you think it is hard to add in general in 1 walk through database and it is worth to consider another extra requests. Please understand the reasoning that extra request could be much slower than doing it in 1 go.

[mmd-osm]

I will leave this one for Roland to take over and add some further comments and insights

[drolbr]

Please have a look at this blog post. You can now get extra context for a diff.

[vshcherb]

Thanks for a reference! So far we solved the original issue in a different way and I will have a talk on SoTM 2018 about this as well. So, probably we could catch up and understand completeness / speed of all methods.