operations
operations copied to clipboard
openstreetmap
Nominatim server reaching capacity
We've had a 50% increase in requests to nominatim.openstreetmap.org in the last year. The result is that dulcy is slowly reaching its limits. To give a rough estimate: it can take another 10-20% increase before it runs out of capacity. As a short term solution I can move pummelzacken back from being a backup server into active production. However, I'd prefer not to make that a permanent setup. We'd be without backup against outage again and maintenance becomes a real pain.
The most simple solution would be to set up a third nominatim server. The new server should have the same specs as dulcy. Only the disks can be a bit smaller (500GB OS disk + 1.2TB DB disk is more than sufficient), Together with pummelzacken, this gives us a setup that should be sufficient for a couple of years.
I've also been toying with the idea of setting up something more distributed. Two options come to mind:
- Simple memcached based caching frontends, maybe even using the tile CDN.
- Partial database replication to smaller servers. This might be most interesting for /reverse requests that only need a small part of the database. The server specs might be small enough that we can ask for sponsored servers similar to the tie CDN servers.
I've tried neither of these options yet and in particular don't know how much work is involved for how much gain.
Together with pummelzacken, this gives us a setup that should be sufficient for a couple of years.
I'd rather go with a setup that has enough disk to last at least four years without upgrades. I'm also not sure going with 1.2TB instead of the 2TB of dulcy will save us much, a small minority of the drive lines have something that capacity instead of 1.6-2TB.
50% growth in a year is quite something. Any idea what's causing this? Is abuse still as big an issue as the last time we talked about it in Karlsruhe?
I imagine the same reason as tiles - people don't want to pay Google's new charges.
I expect the given disk size will do fine for 4 years. Nominatim's DB has actually has decreased in size over the last couple of years from 1.1 to 0.7TB. There is still a bit of room for improvement in the software to keep the database well below 1TB even with the current growth in OSM.
There is one exception: if we want to do search-as-you-type at some point using photon or some other elasticsearch-based solution, I would add another 500GB disk.
@grischard The growth comes from presumed ex-Google users who are mostly small time users which are well within our usage policy.
Out of curiosity, I looked at dulcy‘s munin stats and wonder where this might hit a bottleneck in the near future.
CPU seems somewhat busy (with high system time!) but still below 50% total utilization most of the time. Everything else doesn’t look too bad and should cope with some extra load. Am I missing something obvious here?
The server tends to get clogged under higher load. I never could quite figure out what is the exact reason or even if postgresql or apache or php-fpm are at fault. Things have improved with postgres 10 and the latest changes to the reverse algorithm in Nominatim. It was running smoothly for a while but in December the issue returned for short periods. That's where my capacity estimate comes from.
First of all, it's quite difficult to assess the situation from the outside with only munin stats at hand. So please bear with me if the following is too much of a speculation.
Although not directly related to a clogged server, there's one thing I found a bit odd, namely the high number of forks/s in comparison to any of the other servers.
https://munin.openstreetmap.org/processes-day.html seems to indicate that dulcy runs with 300-400 forks/s, a number that closely matches the number of nominatim requests/minute (~12k-20k) in https://munin.openstreetmap.org/openstreetmap.org/dulcy.openstreetmap.org/nominatim_requests.html.
Is php-fpm really kicking off a new process for each single request, and is this intended? I'm suspecting that this might be one reason for the high system time.
Apologies, if I'm asking the obvious here ... ;)
Now that's embarrassing. Connection pooling was still disabled on the PHP side. As a consequence postgres was forking like mad. Looks better now. That leaves quite a bit more room for growth. Closing for now.
Reopening.
#337 also shows that we could do with a load balancer/CDN in front of the Nominatim service that protects the server from freeloaders.
According to https://munin.openstreetmap.org/openstreetmap.org/dulcy.openstreetmap.org/cpu.html and https://github.com/openstreetmap/operations/issues/337, Nominatim doesn't appear to be overloaded anymore.
Closing again until next time?
No. We need some spare capacity to deal with idiot clients like the one in the #337 incident (who by the way isn't gone, we just manage to hand out all the 403s fast enough now). It is not super-urgent anymore but it should go into the 2020 planning.
A new server is in the 2020 budget. Since this would go in a UK datacenter it's easiest to wait until COVID restrictions ease.
As an ordinary user I have been having approximately 25% success rate calling the API in the last couple of days. One-off queries, trying at all hours of the day. Even the web frontend https://nominatim.openstreetmap.org seems to be down three-quarters of the time.
Not complaining because not paying. Just noting it here.
According to our stats all Nominatim servers are working normally at the moment. Note that we have hard rate-limiting in place on the servers. If you have a script running that constantly goes over the allowed 1 request/s, the server will drop your requests (or more precisely all requests form that IP address) eventually.
@lonvia Certainly not going over any rate limit. But yes, I am sending the requests from a small country in southeast Asia. Are you rate-limiting by region?
No, we rate-limit by IP. The country just decides which server your query goes to. Can you please contact nominatim(at)openstreetmap.org and include your IP address and I'll have a closer look at what's going on.
@lonvia It was a one-time task and I got it done in the end with lots of patience. But good to know this information for the future. Keep up the good work.
@lonvia What's the estimated disk space that a nominatim server will need in 5 years? I had a look at dulcy but disk usage for the 2tb /ssd volume is all over the place
As per comment above, I expect the 2TB to be sufficient for Nominatim itself for the next 5 years. Add 500GB if you want to be on the really safe side. And add another 500GB to leave room for auto-complete.
The recent spike in disk usage comes from a bug in the logging mechanism. I'm looking into that.
Switching to nginx has solved most of the stability issues with Nominatim. Capacity now scales nicely with the available hardware, which makes it a bit easier to give projections.
We currently have an average of 500 req/s with peaks around 750req/s. pummelzacken can do a maximum of 250 req/s. For dulcy I project the maximum capacity around 900 req/s. That means that pummelzacken is nowhere near adequate anymore for a cold spare. We can keep it around as a fallback for the search on the website but it won't be able to take even half the traffic.
I've used stormfly-04 last week as a temporary replacement while reimporting the database on dulcy. That machine is nicely spec'd for a Nominatim server especially since we are now highly CPU-bound. My understanding is that the machine is not really in active use right now, so you might want to consider using it instead of getting a new secondary server for Nominatim. Its other advantage is the location: we could split traffic between it and dulcy and reduce latency for the Americas.
With longma in production we now have plenty of capacity for the foreseeable future.
The remaining question is what happens to pummelzacken. I'd prefer to take it out of production now because it is quite a bit slower to respond than the others and longma can easily take the additional traffic. We could keep it around as a hot spare but there is no urgent need for that either. Any two of the three remaining servers can handle the full traffic if need be.