observability icon indicating copy to clipboard operation
observability copied to clipboard

[BUG] Observability plugin's system index creating is impacting on OpenSearch node starts

Open zane-neo opened this issue 1 year ago • 2 comments

What is the bug? User reported an issue that OpenSearch cluster is not able to spin up even if the issue in the log has been fixed: https://github.com/opensearch-project/OpenSearch/issues/14791, after investigation this is an issue caused by observability's system index creation: https://github.com/opensearch-project/OpenSearch/issues/14791#issuecomment-2381132359.

@andrross has concern on the PR to fix this issue: https://github.com/opensearch-project/OpenSearch/pull/15258, so this issue is to collect more details:

  1. Is this system index a critical one and if it's not created, we should block the cluster from spinning up?
  2. Is it possible to make the index creation a lazy operation until the time it's been used in plugin? Also please chime in the discussion in PR if you have more insights on this issue.

How can one reproduce the bug? Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

What is the expected behavior? A clear and concise description of what you expected to happen.

What is your host/environment?

  • OS: [e.g. iOS]
  • Version [e.g. 22]
  • Plugins

Do you have any screenshots? If applicable, add screenshots to help explain your problem.

Do you have any additional context? Add any other context about the problem.

zane-neo avatar Oct 18 '24 02:10 zane-neo

IMO failing to start up a node on a plugin index creation failure seems very wrong. See https://github.com/opensearch-project/OpenSearch/issues/16340 for a bug that could fatally disable a cluster because of this choice.

dbwiddis avatar Oct 18 '24 05:10 dbwiddis

[Catch All Triage - 1, 2]

dblock avatar Nov 04 '24 17:11 dblock