Consider hosting plans for smaller services

[pnorman]

We've got a number of smaller services that don't occupy a great deal of CPU, RAM, or disk. These are mainly tertiary services like OTRS, IRC, OSMF site, etc. We've been using HP DL360 machines for this for some time now, putting multiple services on machines, but is this still what makes the most sense?

Based on the costs of #282, the G9 machines cost us approximately £700/yr, split halfway between hosting costs (power, space, internet) and amortized machine costs over how long they've lasted for us in the past. I'm not really worried about this cost, I care more about stability and what results in the lowest load on sysadmins.

I know @gravitystorm has some thoughts on this which would start with #169 and build from there.

Any changes are likely to be long-term and shouldn't weigh heavily on purchasing decisions, because machines will still need refreshing and we should be able to find a use for refreshed machines regardless.

[pnorman]

I'm not considering the main DB servers or large DB-based data consumers like nominatim and rendering servers - those have a different set of requirements, are a core part of what we do, and we did some rough evaluations of costs earlier this year.

[gravitystorm]

I know @gravitystorm has some thoughts on this

Yes, I definitely have strong thoughts on this! And you are right, the first step in the long-term plan is to have a resilient storage option, so that we can move the services around between machines without losing state.

In the short term, there's a lot of these services that don't need to be run on OSMF-owned dedicated hardware. Also, we've recently seen how even well-run services interfere with each other when run on shared linux environments (e.g. OSQA blocking system upgrades for other services), so running each service independently might have some advantages too.

Perhaps moving them to hosted VMs (e.g. on Hetzner) is a cheap and easy alternative? Although even with the smallest VMs, they would be more resourced (i.e. more expensive) than many of these services need. Another alternative would be to run them in VMs on hosted hardware, but that trades cost efficiency for extra sysadmin work in maintaining a VM-based platform.

£700 GBP per year would get us around 11 x 2CPU/4GB/40GB VMs from Hetzner, and there's a smaller option too if particular services don't need that much. Or there's a range of dedicated hardware options if we want to rent the hardware and run the VMs ourselves, that removes the overheads of having to organise and own the hardware ourselves.

Of course, I'd prefer that we were doing something more along the lines of resilient storage and disposable containers, but that's taking much longer than I expected to make any progress. So these options are hopefully less work but still useful in the interim.

[pnorman]

£700 GBP per year would get us around 11 x 2CPU/4GB/40GB VMs from Hetzner, and there's a smaller option too if particular services don't need that much.

Note that the costs were per machine per year.

Of course, I'd prefer that we were doing something more along the lines of resilient storage and disposable containers, but that's taking much longer than I expected to make any progress. So these options are hopefully less work but still useful in the interim.

I'd still like to go with resilient storage. I see that as incrementally useful with our current setup. I don't think renting the hardware would be incrementally useful because we'd be swapping managing hardware, which we do, with managing rentals, which we don't have the experience in.

Let's say we had the resilient storage working. How much work are containers on top of that?

[pnorman]

We're not going to go this route. We'd like to better utilize our existing capacity instead, running multiple services on one server.