Monitoring and Alerting

[mudit3774]

Hey,

We are planning to use Tile38 in production. We were wondering what is the best was to set up monitoring and alerting?

Should we rely on system metrics like load averages, CPU, disk metrics etc. only?

Are there any additional metrics that we should monitor? Is there an exporter for Prometheus that we could potentially use?

Please point me to the documentation, if any

[tidwall]

Hi,

The SERVER ext command may be what you're looking for. It returns a bunch of extended server metrics.

There isn't a built-in exporter for Prometheus, but in the past, we've whipped up a custom sidekick that proxies and reformats the metrics into something that Prometheus understands.

[mudit3774]

Thanks for the reply @tidwall

I am sure we can write a custom script but would it make sense to consider something like this: https://github.com/oliver006/redis_exporter

Do you recommend this or have some other recommendations? I am sure you must be aware, it has some tile38 support.

[tidwall]

I haven't used redis_exporter myself, but it looks pretty good. I'd definitely give it a try first.

[epsilon-akshay]

@mudit3774 have you set up monitoring for tile38 yet, are you using Prometheus with a redis exporter?

[tidwall]

I've heard good reports about using Tile38 with redis_exporter.

Also, the https://github.com/tile38/tile38-prometheus repo was released recently open-sourced.

[oliver006]

Hey there, I'm the maintainer of the Prometheus redis_exporter which indeed works for Tile38 (though it's a bit limited with what it can export). I just merged a PR that adds additional metrics that the exporter provides about a Tile38 DB (mostly by pulling data from SERVER EXT).

I see you released a Tile38 sidekick / exporter - but why not add Prometheus support natively to Tile38 itself?

[tidwall]

Hi @oliver006,

It may be worthwhile to support it natively but I'm not very familiar with Prometheus. The Tile38 sidekick was made by a colleague a couple years ago. I would probably need to spend some time investigating.

[oliver006]

Ah ok, that makes sense, Josh. I might have time in the coming week or two to put together a quick PR that shows how to expose some basic metrics.

[tidwall]

That would be really helpful!

[oliver006]

Ok, great. I'll put something together and then we can figure out how to get some more useful metrics exposed later, once we have the skeleton.

[oliver006]

I opened #604 to expose some basic metrics - let me know what you think.