perses
perses copied to clipboard
perses
Support ClickHouse as an observability backend
Proposal
We see more and more adoption of ClickHouse as an observability backend that can efficiently deal with very large datasets with good support for OTel. At the moment, it's best suited for logging and tracing support but ClickHouse has on the roadmap for this year a PromQL compatibility layer that will enable the metrics use-case as well.
Here are some references:
- OLAP for Monitoring with ClickHouse on Kubernetes, at eBay
- HTTP Analytics for 6M requests per second using ClickHouse, at Cloudflare
- A cost-effective logging platform using Clickhouse for petabyte scale, at Zomato
- Fast and Reliable Schema-Agnostic Log Analytics Platform, at Uber
- ClickHouse for Observability, at Gitlab
- LLM Monitoring, at Helicone
- ClickHouse for OpenTelemetry Traces, at Resmo
ClickHouse is also used as a backend for some popular observability SaaS providers including:
Disclaimer: I work at ClickHouse and we will be happy to help!
Hi @Ryado,
It's really cool to see observability backend like ClickHouse that would like to support Perses !!
I think it can make sense that Perses can support other thing than just Prometheus or Tempo. For example time to time, people would like to connect Perses to a mysql database.
But before taking time to adopt others datasources, I think we really need to work on the plugin part. Today implementing a plugin is quite hard and it needs to be done in the upstream repository because it needs to be compiled with everything else.
It becomes more and more urgent to be able to externalise the plugin and so to be able to load a plugin at the runtime. If you want to help on this area, that would be super cool ! Once it is done, we can then consider if adding a ClickHouse plugin directly in Perses makes sense or if you should just provide it on your own side.
Makes total sense. Please let us know when the plugin architecture is ready and we can definitely consider a contribution to support ClickHouse
Hey @Nexucis, very interested if there's any progress on the plugin integration. Perhaps we could even implement this in the upstream repository if the internal API is stable enough. Will you be at KubeCon NA 2024? I'm in the CNCF Slack too if you have time to chat about this more.
Thanks!
Hey @SpencerTorres, yeah we are working on it, slowly, but it's progressing. We started to externalise the panel as you can see in https://github.com/perses/perses-plugins. Others plugins will follow. We have some difficulties to externalise the datasource, but it will come for sure.
I won't be at KubeCon NA 2024, but I should be at KubeCon EU 2025. I am also available on the CNCF slack. You can contact me there whenever you want. (Just in case, I will be off tomorrow until the 20th October.
Hey @SpencerTorres I am back from holidays and available to talk if you still want to talk ofc :)