Alert message: "osmDiff: runtime error: Query run out of ..."

[pa5cal]

I'm not sure, if this is the right repository, but lately achavi reports/displays in many cases alert messages such as: "osmDiff: runtime error: Query run out of memory using about 2048 MB of RAM."

Example: https://overpass-api.de/achavi/?changeset=62679580

Maybe it's the result of https://github.com/nrenner/achavi/pull/39 ?

[nrenner]

Yes.

The error message is taken from the Overpass API server response, in this case:

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.55.4 3079d8ea">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2018-09-18T06:43:03Z"/>

<remark> runtime error: Query run out of memory using about 2048 MB of RAM. </remark>

</osm>

https://overpass-api.de/api/interpreter?data=[adiff:%222018-09-17T23:05:11Z%22,%222018-09-17T23:05:16Z%22];(node(bbox)(changed);way(bbox)(changed););out%20meta%20geom(bbox);&bbox=-118.006896,33.7793753,5.6256223,53.4836593

[nrenner]

Changesets with a large bounding box like this one are a known issue, see #9 and also drolbr/Overpass-API#322.

OSMCha has better support for those, as changesets are compiled and stored from minutely augmented diffs: https://osmcha.mapbox.com/changesets/62679580/?filters={"ids":[{"label":"62679580","value":"62679580"}],"date__gte":[{"label":"","value":""}]}

Beside now showing an error message where it failed silently before, I'm not aware of any functional changes that could cause more errors.

[mmd-osm]

By the way, the changeset API call now also includes the number of changes:

https://www.openstreetmap.org/api/0.6/changeset/62679580.xml

This way, we have one more decision point to switch to an "id" based approach for changesets with a large bbox and a relatively small number of objects, e.g. download the changeset via the API, extract the object ids and use those in the attic query.

(I added the ".xml" so that the call get directed to the Rails ports, instead of cgimap, as cgimap 0.6.2 hasn't been deployed yet to production).

[pa5cal]

Just want to mention that the changeset dump and replications files already contain the "num_changes" attribute.

[mmd-osm]

Yes, that's exactly the same information. It's called changes_count though, to have it more in line with the existing comments_count attribute.

We discussed the "id" based approach at length in #9, but it was never clear, if downloading the xml changeset via API is prohibitively expensive or still ok.

[mmd-osm]

Another option to deal with huge bounding boxes and very close begin/end timestamps could be to use a global query without any restriction, and to filter out the relevant transaction number afterwards. ~As an additional safeguard, the total number of objects can also be restricted.~ (doesn't work this way)

In this particular case, the query is extremely fast, taking less than 1s.

[adiff:"2018-09-17T23:05:11Z","2018-09-17T23:05:16Z"];
(
  node(changed);
  way(changed);
);
out meta geom;

[nrenner]

By the way, the changeset API call now also includes the number of changes

Nice, I had thought about requesting this, might also be helpful as

• additional indicator for overlapping changes
• check if changesets queried without potential timeout (- 60 min) are complete

Another option to deal with huge bounding boxes and very close begin/end timestamps could be to use a global query without any restriction

Yes, the question then is up until what limits to use which query. Some time ago I started testing query variants with and without bbox and/or changed for different zooms and time ranges, but didn't finish.

I also wondered whether this kind of query optimization should better done on the server?

I was sort of hoping that OSMCha could be the successor for changeset visualization, but it seems that they also gave up on covering all the edge cases of using adiff to query changesets. An option would be to help fixing the remaining issues there.

Before doing further work on this, I guess I would want to take a step back and doing a writeup on all the issues and possible additional datasources and options that could help, or what potential alternatives to Overpass API there might be for getting changeset data.

[mmd-osm]

I was sort of hoping that OSMCha could be the successor for changeset visualization, but it seems that they also gave up on covering all the edge cases of using adiff to query changesets. An option would be to help fixing the remaining issues there.

I think it's even worse. All devs, except for WIlle seem to be gone in the meantime.

[nrenner]

I got that impression, too.

And now that they have developed their own internal review pipeline, the future of OSMCha seems even more unclear.

[mmd-osm]

Yes, the question then is up until what limits to use which query. Some time ago I started testing query variants with and without bbox and/or changed for different zooms and time ranges, but didn't finish.

I also wondered whether this kind of query optimization should better done on the server?

Yeah, I don't think it makes sense to to this on the client. Let's see if we can move changeset based (changed) filtering into production next year, which should solve more or less all of achavi's current issues.