kibana icon indicating copy to clipboard operation
kibana copied to clipboard

Published 20 hours ago • verified publisher icon elastic

[Stack Monitoring] Support for integrations

Open neptunian opened this issue 3 years ago • 1 comments

Summary

The Stack Monitoring-supported integration packages (elasticsearch, kibana, logstash) are currently in an experimental state. While Stack Monitoring added support to read from the datastreams created by these integrations in 8.1, there are remaining issues that prevents the packages from being an official source of metrics powering the SM UI.

This issue tracks the effort to bring these integration packages to a GA state, providing feature parity with the 8.x beats modules running in standalone mode in the Stack Monitoring UI.

Useful links:

Utils

  • [x] https://github.com/elastic/integrations/issues/3849
  • [x] https://github.com/elastic/kibana/issues/138052
  • [x] https://github.com/elastic/integrations/issues/4061

Packages

  • [x] https://github.com/elastic/kibana/issues/135379
  • [x] https://github.com/elastic/integrations/issues/3929
  • [x] https://github.com/elastic/integrations/issues/4011
  • [ ] https://github.com/elastic/kibana/issues/137112
  • [ ] #138865
  • [ ] https://github.com/elastic/integrations/issues/3974
  • [ ] https://github.com/elastic/integrations/issues/4107
  • [ ] https://github.com/elastic/integrations/issues/4187
  • [ ] https://github.com/elastic/integrations/issues/3917

Stack Monitoring

  • [x] https://github.com/elastic/kibana/issues/104271
  • [x] https://github.com/elastic/kibana/issues/120416
  • [x] #138864
  • [x] https://github.com/elastic/kibana/issues/135382
  • [x] https://github.com/elastic/kibana/issues/140039
  • [x] https://github.com/elastic/kibana/issues/123508
  • [ ] https://github.com/elastic/kibana/issues/140358
  • [ ] https://github.com/elastic/kibana/issues/123869
  • [ ] #141305

Testing

  • [ ] https://github.com/elastic/integrations/issues/4008
  • [ ] https://github.com/elastic/integrations/issues/4013
  • [ ] https://github.com/elastic/observability-dev/issues/1660
  • [ ] #137195

Acceptance criteria (to be expanded):

  • Stack Monitoring UI is enable-able by installing elastic-agent and the Elasticsearch/Kibana/Logstash packages
  • Stack Monitoring UI and alerts behave similarly when reading data from metricbeat or from the packages
  • Troubleshooting documentation exists to provide guidance in onprem/cloud issues
  • A dashboard provides insights in usage

Follow ups:

  • [ ] https://github.com/elastic/observability-dev/issues/2241 : Once integrations are supported, we'll have 3 collection modes (internal, standalone, agent). This should allow us to move forward with deprecation and eventual removal of internal collection.
  • [ ] https://github.com/elastic/kibana/issues/120414
  • [ ] https://github.com/elastic/integrations/issues/4116

neptunian avatar Dec 03 '21 21:12 neptunian

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

elasticmachine avatar Dec 03 '21 21:12 elasticmachine

@crespocarlos @klacabane I was just noticing we don't have integrations (I think) that would cover:

  • beats
  • apm server (integration server)
  • enterprise search

Those 3 components can be activated in the Stack Monitoring UI, so I'm not sure we can recommend migrating to agent/package based until we have packages for those.

matschaffer avatar Oct 06 '22 05:10 matschaffer

Yeah. Fair point. I don't think we have either.

I'm wondering if we could make Kibana, Logstash and ES packages production ready and start with them - perhaps they can be tagged as beta for now. The SM UI will work with both methods anyways. We can't surely recommend phasing out metricbeat for SM without having all the packages, but users could start using the packages.

The remaining packages will gradually be released and, once we're done, we'll change the documentation to recommend the migration.

crespocarlos avatar Oct 06 '22 07:10 crespocarlos

Per discussion yesterday, beats and apm-server might be covered by the agent package. In an agent-based setup we probably don't need a beats monitoring package.

So that'd just leave enterprise search on the list of things that Stack Monitoring supports today that we don't have observability packages for.

matschaffer avatar Oct 11 '22 02:10 matschaffer

Does that also apply for APM Server running outside of Elastic Agent?

miltonhultgren avatar Oct 11 '22 07:10 miltonhultgren

What I got from @klacabane is that we might be able to assume standalone apm server doesn't exist in the agent/package-based world.

Current stack-monitoring for standalone apm server can continue to be done using standalone metricbeat (or probably even internal collection for some time).

matschaffer avatar Oct 12 '22 02:10 matschaffer

I think this boils down to how we want the transition to agent to happen and if we want the SM experience to remain the same as a first step of the migration, or if we're fine moving into integration territory earlier and breach the parity milestone that this ticket aims at. For elasticsearch/kibana/logstash the answer is yes, and I would add enterprise search to that list. The reason is that we have clear plans for these with Platform Observability. For beats and apm server I'm not sure:

  • Beats is an implementation detail in agent world and I don't think we want the agent abstraction to be leaky, mostly because it can confuse users and beats will be eventually deprecated so this is a good opportunity to already get rid of the concept. So here what we could do is to detect whether metrics are collected by agent and offer a redirection to the Agent integration (maybe a dashboard if we have a reference) under the stack monitoring Beats section
  • in both standalone/fleet cases the Apm integration needs to be installed so any dashboards that it contains will be available. We can have a mechanism similar to the beats one and take the opportunity to jump over the SM parity step. Alternatively the parity step could probably be achieved by adding the apm indices in stack monitoring if the documents are similarly shaped, so that's low effort and I'm fine with that as well

klacabane avatar Oct 12 '22 10:10 klacabane

~After looking at apm-server it looks like the metrics are not automatically collected by agent when running as a subprocess and needs a separate metricbeat to collect them.~ For the beats subprocesses, the metrics are collected at metrics-elastic_agent.(metricbeat|filebeat)-*. One thing I forgot is that beats are not only filebeat and metricbeat spawned by agent but could also be independant heartbeat, auditbeat.. that may not have a corresponding entry in the agent catalog so we should still be able to monitor these with agent.

So for beats, including apm-server, it looks like we need two things:

  • read from metrics-elastic_agent.*beat-* to surface metricbeat and filebeat spawned by agent
  • create a beats package that collects metrics of independant beats and apm-server

We could also investigate if agent would be able to collect apm-server metrics itself just like it does for the beats

Digging a bit further I see references of metrics-elastic_agent.apm_server-*, metrics-elastic_agent.heartbeat-* ... so everything should already be there and it would just be a matter of adding these indices to the right places in Stack Monitoring

klacabane avatar Oct 13 '22 12:10 klacabane

Closing as all tasks here are complete. Follow ups will be created directly on the board

klacabane avatar Apr 12 '23 13:04 klacabane