Host OpenStreetMap Spyglass

[pnorman]

After @joto's talk on OpenStreetMap Spyglass the general consensus was we would like to host this as part of a replacement of the map call data layer on osm.org.

The osm.org website map view needs to be moved to Maplibre GL for this to happen, but we should get Spyglass running in parallel. This will let us gather operational experience before the website depends on it.

Hardware requirements

Disk

1.5TB is needed for database creation. Growth is likely to be similar to other osm2pgsql databases

RAM

joto's server runs with 128GB of RAM.

CPU

Unknown, but joto is running it on a relatively unpowerful hetzner machine

The suggestion last night was to run it on the overpass machine. We also discussed the potential of replacing the query feature with something powered by the spyglass DB

[Firefishy]

Code: https://codeberg.org/jot/osm-spyglass

[pnorman]

I've unassigned this because we identified this as a good task for getting more people involved with ops.

[pnorman]

I posted to the community forum about looking for sysadmin help for this

[mmd-osm]

So this looks like it has a fair amount of overlap with what's already there in https://github.com/openstreetmap/chef/tree/master/cookbooks/vectortile, with a custom go based tile server instead of tilekiln.

Building binaries isn't covered there, though there are other examples around, like https://github.com/openstreetmap/chef/blob/e109384b70bcec3d57db75f8f2c058762ae627ae/cookbooks/db/recipes/base.rb#L43

Is the demo website also in scope, or just the backend service?

Anything from the talk that's worth mentioning here? It's still not available online as of now.

(No, I don't have the time to look into this in more detail)

[joto]

Is the demo website also in scope, or just the backend service?

That's not just a demo website, the backend service and the website are intended to work together. This is not (yet) a general service. Later on this might develop to be a more general service, but at the moment that backend is made specifically for this site only.

[mmd-osm]

Well, I thought the ultimate goal was to "host this as part of a replacement of the map call data layer on osm.org.". Somehow spyglass needs to be integrated into openstreeetmap-website, although I'm not clear how this would work. If this has been discussed already, it would be good to document it.

[joto]

No, this has not been discussed at all. At the moment Spyglass is its own thing and should be seen as that. A "replacement of the map call data layer" can mean many things, full integration into the rest of the map on openstreetmap.org is only one option. At the very least, Spyglass needs Maplibre, so that's not something which is going to happen any time soon. Lets not waste our time discussing what might be step 17 in a long process, lets keep this to the next step which is getting Spyglass online as it is.

[mmd-osm]

Thanks for clarifying. It wasn't entirely clear from @pnorman's initial post what was planned here.

By the way, Maplibre is already in production since last week (https://github.com/openstreetmap/openstreetmap-website/pull/6504). I'm expecting the main map to follow fairly soon as well. From a planning perspective, the current leaflet map data layer would have to be ported to Maplibre. The rest is then open for discussion.

[mmd-osm]

So here's a very quick and dirty version running in Chef (locally). nginx and go server are up and running, there's a tiny bit of data imported in Postgresql to show a few nodes on the screen.

I've uploaded the changes here, if anyone is interested https://github.com/mmd-osm/chef/tree/feature/spyglass It's not very likely that I'll create a pull request for it. If someone wants to take it from here, feel free to do so.

Some open topics:

• Go requires a version that's available from bookworm-backports. The original bookworm version 1.19.* was too old. I had to add -buildvcs=false to bypass some build error. It would be good to have a clear step-by-step description how to build the go server on Debian Bookworm.
• The import was complaining about some constraints that could not be set. Might be a permission issue, or caused by the PBF sample data containing nodes only. I haven't investigated this in more detail.
• Import script was tweaked manually to disable later updates due to the large memory requirements.
• Raster doesn't seem to work (go server responds with HTTP 500). Might be caused by import issues above.
• Update script was not tested.
• Updating the hostname in web/map.js was done manually in Docker, and it not yet automated. I wasn't sure how this supposed to work in the future. Patching web/map.js each time? Moving the hostname to some other file, and patch that by Chef? Assume the current hostname as default, and get rid of the hostname variable?
• Import and update scripts have been moved to Chef, rather than keeping the version from the spyglass repo.
• ~Running ALTER DATABASE osm SET postgis.gdal_enabled_drivers TO 'PNG'; was also done manually. It's not clear how to do this from recipes/default.rb~
• Anything user/authorization/database name specific is still a huge mess, same as putting Git repo contents, etc. in a proper filesystem structure.
• I changed the database name from osm to spyglass (osm is too generic for osmf infra).
• ~Tests probably don't work either (not tested)~ Github Action tests were in fact passing. However, I reduced the scope of what was being tested a bit.

Edit: with a larger extract that includes nodes, ways and relations, the raster layer is also visible:

Query

[out:pbf][timeout:300][bbox:{{bbox}}];
(node({{bbox}});way(bn);node(w););
(._;(rel(bn)->.a;rel(bw)->.a;);rel(br););
out meta;

[joto]

Go requires a version that's available from bookworm-backports.

Options I see are:

• Trying to fiddle with this to maybe get it to work with older go compiler. No idea how realistic this is and, even if possible, this will likely break sooner or later.
• Use backports. Sooner or later we'll switch to Trixie for the server anyway which will solve this problem. (At least for a while.)
• Compile somewhere else and install the binary. Maybe use the CI for that. (Although the code is not on Github, so there is currently no access to a CI runner.)

The import was complaining about some constraints that could not be set.

This is related to the raster creation. Some of these are "normal" and can be ignored, I have not found out yet what the problem here is exactly. But some of them are due to you not importing the whole planet and that's also what's breaking the raster stuff. I have seen that problem before when developing. It would be good to get this to be more robust, but I am not sure yet how.

Updating the hostname [...]

I can see two useful options for this:

• Use the current hostname by default, i.e. use relative URLs, allow to overwrite this in the script. Means it is easy to install if you install the server at the same time.
• The spyglass.openstreetmap.org as default, which means it will work for our use case here and it will also work for everybody who installs just the UI. Makes it easier for people who want to work just on the UI.

I changed the database name from osm to spyglass (osm is too generic for osmf infra).

That's probably something I can do upstream.

[pnorman]

We've put out a call for help on this project as part of an effort to get new sysadmins involved in ops: https://community.openstreetmap.org/t/looking-for-sysadmin-help-with-new-openstreetmap-spyglass-service/138487

[mmd-osm]

But some of them are due to you not importing the whole planet and that's also what's breaking the raster stuff. I have seen that problem before when developing. It would be good to get this to be more robust, but I am not sure yet how.

I'm getting error messages like the following in the systemd service log:

Dec 22 13:21:35 dokken spyglass[13446]: ERR 0xc0003d6780 RASTER raster/relations/6/38/19 QueryRow failed: no rows in result set
Dec 22 13:21:35 dokken spyglass[13446]: ERR 0xc000357400 RASTER raster/relations/6/38/22 QueryRow failed: no rows in result set

It seems you need to add some special case handling for an empty result set so that it isn't considered an error anymore. According to https://github.com/jackc/pgx/issues/677#issuecomment-2713945524 the respective value should be pgx.ErrNoRows.

I was wondering a bit about the 64 bit hex value between "ERR" and "RASTER". This seems to be the request memory location. Do you need it for debugging purposes?

[mmd-osm]

We've put out a call for help on this project as part of an effort to get new sysadmins involved in ops

There's a reason why I wrote "It's not very likely that I'll create a pull request for it. If someone wants to take it from here, feel free to do so."

This proof of concept was, so to speak, the very first iteration to see what changes are needed on the Spyglass repository and how to get the whole thing up and running under Chef control. The real work is only just beginning.

[joto]

It seems you need to add some special case handling for an empty result set so that it isn't considered an error anymore.

Fixed now.