Update lookups

[afischerdev]

This is the call for a bigger update on lookups.dat

The Android apk doesn't need an update for that. Lookups.dat and profiles are updated automaticly. Next apk update will then contain the new versions.

So only web services are affected.

And an update for the profiles is needed when there is an interest on man_made=pier or other changes.

Please see #416 for more.

[zod]

If we generate new segments using this update lookups.dat all users must update their segment files as soon as they get the new lookup.dat. Correct?

• I'm not to sure about compatibility. When do we need to increase the major version?
• I don't think we need hiking-beta.brf in the repository, but just as downloadable profile.

[afischerdev]

If we generate new segments using this update lookups.dat all users must update their segment files as soon as they get the new lookup.dat. Correct?

Yes. When the server update is done and an apk user updates the data lookups.dat and profiles are downloaded first then rd5 download starts. But when user has called e.g. only one rd5 but has 4 on board he has a problem when next enter an outdated rd5 - he will get a notice. We already discussed if we drop the others or not.

I'm not to sure about compatibility. When do we need to increase the major version?

When index changes a new number is needed. Smaller changes like aliases don't change it.

I don't think we need hiking-beta.brf in the repository, but just as downloadable profile.

Yes, until new apk version is out. Current looks for hiking-beta But we could exclude it from profile zip

[polyscias]

I think breaking things for (many) users is bad and like I wrote in #416, I see quite some pain for removing oneway=true.

I did post PR to update the profiles of @poutnik, see poutnikl/Trekking-Poutnik/pull/30 but that has not been merged as of now.

Apart from the profiles of @poutnik it also impacts most other profiles so I think it is better to keep it in the profile for now, but let's remove it for the brouter profiles.

But when user has called e.g. only one rd5 but has 4 on board he has a problem when next enter an outdated rd5 - he will get a notice.

How does that notice look like? I run my tests with the server but there the error reporting is not good, the error is only logged at the server side.

Also, what about return meta.lookupVersion == 10;

For the remainder the patch looks good to me, I can run later this weekend a map generation run with it and give some stats.

[nrenner]

Quoting @polyscias from #416 (comment):

For tracktype=4, can we added "grade3-5" as alias, that is mapped 917 times

I wondered where the tracktype=grade3-5 value comes from.

The OSM tag history graph (see also Taginfo chronology) looks suspicious:

And indeed, there are two unique uses:

• 945 ways added by an import in 2014 with changeset 21660337 for some region in Norway [1]
• 538 ways added since July by a single paid mapper from Kaart [2], almost all to highway=residential in South Africa

The latter and other sporadic edits are all using the iD editor, which actually has that value in the list of suggestions when using the properties table, which is most probably because of the value count of that single import.

Therefore I think the grade3-5 alias should not be included, such an endorsement would only further propagate the use of this undocumented and unsupported value.

---

[1]

curl -o changeset-21660337.osc https://www.openstreetmap.org/api/0.6/changeset/21660337/download
grep "grade3-5" changeset-21660337.osc | wc -l

[2] https://overpass-turbo.eu/s/1kNX

[polyscias]

Nice analysis @nrenner!

I am perfectly fine with removing the grade3-5 alias but my opinion is, also with knowing this story, such an alias is fine.

There are more tags in lookups.dat that are undocumented, for instance noexit=yes in way context and junction=approach

[poutnikl]

I did post PR to update the profiles of @poutnik, see poutnikl/Trekking-Poutnik/pull/30 but that has not been merged as of now.

I have somehow missed the PR and related discussion. I will try to address it and generate new profiles, but after Aug 15, due vacation.

Another delay will be for LocusMap users, with profiles embedded to application or online LocusMap client. They use profiles from several sources.

[polyscias]

I have somehow missed the PR and related discussion. I will try to address it and generate new profiles, but after Aug 15, due vacation.

That would be nice.

Another delay will be for LocusMap users, with profiles embedded to application or online LocusMap client. They use profiles from several sources.

Do you have some more information on this, in particular the exact profiles that are used? I like to add them to my collection of profile that I check for impact of this update.

I can run later this weekend a map generation run with it and give some stats.

With quite some delay I did run it and looking at all the .rd5 files generated versus those on http://brouter.de/brouter/segments4/ I see a mix of segments that did increase in size and decrease in size but net effect is that the overall size is -0.04%

The extremes:

W90_N30.rd5   4998766   34854799 +14.34%
E25_S10.rd5    477592    3252900 +14.68%
E45_S30.rd5    113831     768699 +14.81%
..
E10_N65.rd5   -505069    3792746 -13.32%
E15_N65.rd5   -952119    7259065 -13.12%
E10_N60.rd5  -3225188   23634055 -13.65%

I am planning to get the updated lookup.dat, profiles and one .rd5 on my mobile and see what it does.

[polyscias]

See also brouter-web#669, can we add also:

• railway=platform
• public_transport=platform
• man_made=pier

The first two are accessible on foot and a pier is often a way to access a ferry both on foot and by bicycle.

[afischerdev]

@polyscias All your wishes are already done:

• railway=platform

update

• public_transport=platform

update

• man_made=pier

update

[afischerdev]

@abrensch @nrenner Are there any timelines for publishing this update?

[afischerdev]

Thanks for the reply.

Do you really think it's neccessary to change the major lookup version?

I'm not sure. My understanding until now was the minor version is for updates without index changes e.g. new alias values. Major version is for changes on the index. And that is what we have with this update.

current apk: It downloads lookup file first, so a check for major version should work in download manager.

web client: should be easy

I thought the new data are generated into a new folder first and by rename it (and publish the new lookup and its standard profile) the switch is done. Only problem I see is: the other profiles needs an update too.

[abrensch]

My understanding until now was the minor version is for updates without index changes e.g. new alias values. Major version is for changes on the index. And that is what we have with this update.

adding values and adding tags are also changes compatible with minor updates

I see that lookups.dat from this PR is not minor compatible, e.g. due to the deletion of lines who's values as been aliased

But instead of deleting these lines we could just place dummy-values as placeholders

Opening Pandoras Box by sending incompatible RD5's into the world is something that needs more careful attention and engeneering, compared to making a lookups.dat patch minor-compatible

I will have a closer look but that wil take sone days

[polyscias]

But instead of deleting these lines we could just place dummy-values as placeholders

I did some analysis of all profiles I could find, see https://github.com/abrensch/brouter/issues/416#issuecomment-1100707613, I think non-used values that are not used by these profiles can be safely removed without triggering compatibility issues.

I made a patch for the profiles of @poutnikl but that has not been merged yet (no blame), it shows that deleting certain unused values will indeed give compatibility issues and I think it is not worth that pain so I agree we better place dummy-values as placeholders for these particular values.

It would mean to retain (as dummy-values):

• oneway=true
• sac_scale=T1-hiking
• highway=toll_bridge
• highway=city_entry

I did leave out the keys for https://github.com/utack/utack_brouter_profile but created https://github.com/utack/utack_brouter_profile/pull/35

[afischerdev]

I add a lookup without major change All deleted lines have a dummy for the index All new values are at the end of section.

For testing: lookups_no_major.dat

[afischerdev]

What are the blocker to publish a new major lookups.dat?

I don't see problem on the web clients. They will find new lookups.dat and new generated tile in segment folder.

The Android app could be the problem. I see some constellations which could be discussed with major or minor update.

1. situation: user update all in one step - great function in BRouter ;-)

• major: download of the new lookups.dat (and profiles), download all new tiles as a whole update
• minor: download of the new lookups.dat, download tiles. When old tiles are outdated a whole update as well, on a near by update before the RD5DiffManager should do the job.

2. situation: user update only one tile

• major: download of the new lookups.dat and one tile result: one tile is working, call for the other tiles will run into an error 'lookup version mismatch' - something like 'datafile not found', user has to download other tiles
• minor: download of the new lookups.dat and one tile result: one tile is working, call for other tiles will not run into an error, but delivers unexpected results, user has to download other tiles.

3. situation: user has handmade profiles

• major: download new lookups.dat ... when the profiles uses parameters that are not present in the new lookup it will end in an error
• minor: download new lookups.dat ... when the profiles uses parameters that are moved to dummy, it will break.

More ideas?

[polyscias]

I had a look at lookups_no_major.dat, it looks nice to me. Let me run in the weekend a rd5 creation test with it.

Based on the brouter_route_placeholder_dummy's I understand the order of key-values in the lookups.dat is how things are encoded.

What I am wondering is if it is not possible to replace "deleted" key-values that are now converted to brouter_route_placeholder_dummy's to new key-values. That would work for:

• highway, bicycle, motorcar and motorcycle and 1 key-value for vehicle in way context.
• highway, barrier and entrance in node context

Also brouter_route_placeholder_dummy 59..80 are not needed to my opinion.

I think, to prevent breaking profiles, oneway=true, sac_scale=T1-hiking, highway=toll_bridge and highway=city_entry should be retained (no brouter_route_placeholder_dummy) although their usage in OSM is zero or almost zero.

On the Android App: I think that the download manager should be updated so it detects that there is a new lookups.dat and if so gives the (as only) option to update lookups.dat and all tiles. That will prevent problems.

I am not worried about handmade profiles, like I wrote I did check all public available profiles I could find and no problems apart from those 4 mentioned key-values. I do think there are almost no or zero persons that have not published their handmade profile and are using keys that are deleted in this lookups_no_major.dat

[afischerdev]

@polyscias Ok I add the breaking entries again.

[polyscias]

I dd run mapcreation on a looksup.dat based on lookups_no_major.dat but with my proposed changes above.

It looks good, the total size of all .rd5's did go from 7878.0 MB to 7877.9 MB so a very small decrease in size.

On compatibility:

• Is there any way to test if a new lookups.dat is compatible with the older?
• Also if there is no update of the major version number I think it is bad if a user is running with a mix of old and new .rd5's.

[afischerdev]

Is there any way to test if a new lookups.dat is compatible with the older?

I would think we could test a new lookup against the standard profiles.

Also if there is no update of the major version number I think it is bad if a user is running with a mix of old and new .rd5's.

I would also vote for the major change.

[polyscias]

What I think would be ideal is if the App was updated that once people open the App to download/update segments, it detects there is a new lookup version, and pops up a screen explaining there is a of change the segment format.

Then it asks if the user wants to update and if so, all existing segments will be deleted and have to downloaded again by the user. Optionally the App can offer to re-download all existing segments itself.

I think that is relatively easy to implement but as of now the App is not working this way and also if the App is changed say in the next month, it will not be that if this lookups update in 2 months everybody's App is updated.

In that sense, if we do not want to wait long with this lookup update, it is probably better to go with a compatible update.

I think that in the same time the App should be updated so in the future an update as described above can be done.

I use also QMapShack with brouter integration, I am wondering how that handles lookup updates, let me have a look at that.

[abrensch]

* Also if there is no update of the major version number I think it is bad if a user is running with a mix of old and new .rd5's.

It's also bad without a lookup change to mix RD5s from different preprocessor runs

It was never supported, and in the past there were actual exceptions from that constellation.

Chances are that I mitigated them with my patch for the bee-line hack. Look at the change to OsmPath.java from this commit:

https://github.com/abrensch/brouter/commit/eb716506fd934989177e8c3933b22c58b5a41574

The condition that I changed here from an exception to a "do nothing" is when someone changes a way direction of a way crossing a RD5-tile-boundary and using old/new files at that boundary. So there's no exception anymore but it's still wrong..

Technical background is that the tagging ("description") is encoded at "forward"-links only. But in that constellation a link is considered "backward" at both terminating nodes, so the tagging is not encoded anywhere.

But also for the more likely case that for a crossing link a terminating node is shifted to a slightly different position that link will just disappear, which is also wrong.

But in my opionion that status quo of just accepting that sort of "glitches" for the (unsupported) case of mixing RD5s from different preprocessor runs is o.k. It's better then detecting these conditions and delaing with the support cases.

[afischerdev]

I am not worried about handmade profiles, like I wrote I did check all public available profiles I could find and no problems apart from those 4 mentioned key-values. I do think there are almost no or zero persons that have not published their handmade profile and are using keys that are deleted in this lookups_no_major.dat

It is possible to check the lookups.dat to the used profiles like this: java -cp ../lib/brouter-1.6.3-all.jar btools.expressions.IntegrityCheckProfile profiles2/lookups.dat profiles2

You will need the last master or the lib from git action

[quaelnix]

I also think that we should not support false tagging practices like this:

• tracktype=grade3-5
• highway=traffic_signals;crossing
• traffic_sign:direction=forward;backward
• traffic_signals:direction=forward;backward

[polyscias]

I also think that we should not support false tagging practices like this:

* [tracktype=grade3-5](https://taginfo.openstreetmap.org/tags/tracktype=grade3-5)
* [highway=traffic_signals;crossing](https://taginfo.openstreetmap.org/tags/highway=traffic_signals%3Bcrossing)
* [traffic_sign:direction=forward;backward](https://taginfo.openstreetmap.org/tags/traffic_sign%3Adirection=backward%3Bforward)
* [traffic_signals:direction=forward;backward](https://taginfo.openstreetmap.org/tags/traffic_signals%3Adirection=backward%3Bforward)

For tracktype=grade3-5 see also and earlier comment, meanwhile the usage is higher.

Yes, these values are not documented within OSM but that is an OSM problem and that problem should be solved in OSM.

I added those value aliases based the usage, for brouter the goal is to support as much the data that is present.

[quaelnix]

for brouter the goal is to support as much the data that is present.

I think that's a legitimate point, and I agree that the last three of my examples do little to no harm, but I'm still not convinced that making grade3-5 an alias of grade4 would be a wise idea.

[quaelnix]

Adding kerb=* to the lookup table would allow us to determine an appropriate penalty for barrier=kerb.

[afischerdev]

To get this PR a little bit forward I made a test with an new app to handle the described problems on a version change.

It notified a version change and offer a select box

It also allow only fully rd5 on version change. What do you think?

[polyscias]

Yes, it would be good to get this PR forward and such a Version Problem message is exactly what is needed. One suggestion: Instead of the word "data" I would use the word "tiles"

Apart from this I think we need a way to force a version update of the App. What now can happen is that the tile format is updated but the App is still old so this Version Problem screen will not be displayed and bad things will happen.

I did a bit of thinking on this and I think it would be good to add a "minappversion" to the lookups.dat file like:

---lookupversion:10
---minorversion:14
---minappversion:1.6.3
 
---context:way

With an update initiated through the App, the code should first download the new lookups.dat and check if it's the version is larger than the minappversion, if not, it should indicate the App should be updated and the update process should be exited.

[polyscias]

Adding kerb=* to the lookup table would allow us to determine an appropriate penalty for barrier=kerb.

The current lookups.dat has already wheelchair=* and in that light adding kerb=* would make sense I think.

kerb;0000000001 lowered flush;lowered lowered_and_sloped
kerb;0000000001 flush no none
kerb;0000000001 raised yes regular normal
kerb;0000000001 raised

[afischerdev]

@polyscias Thanks for the reply. The wording for data is changed in my preview.

With an update initiated through the App, the code should first download the new lookups.dat and check if it's the version is larger than the minappversion, if not, it should indicate the App should be updated and the update process should be exited.

The next app could have the lookup version check before download. So I'm not sure if we need an app check as well. The problem is that we can only change the lookups.dat when a new app is online. Otherwise it is possible to publish a new lookup table now but this runs into the described trouble