[docs] Unify internal observability documentation

[mx-psi]

Currently on this repository internal observability is documented in three different places:

• The troubleshooting document includes information about configuring metrics and logs for gathering observability.
• The observability document includes information about how to configure metrics with the upcoming configuration as well as the goals for observability
• The monitoring document includes information about how to use observability metrics for monitoring the Collector

I think we should have a single document that describes:

• How to enable observability
• What is our observability offering and what each metric means
• How to effectively use this telemetry to monitor the Collector

We may want to move this documentation to opentelemetry.io instead of having it here. I would like to have the opinion of @open-telemetry/docs-approvers about this.

[svrnm]

Same here, this belongs on the website from my point of view (expect any pieces that might be more relevant for developers). cc @theletterf is currently working on some improvements on the collector docs

[theletterf]

I think most conceptual information and prescriptive guidance (tutorials, for example) should live in OpenTelemetry.io to reduce fragmentation. There's still room for development and reference documentation on the repos, though also ref could be automated into the docs.

All this to say: I'm in for moving this to the docs.

[kumarankit999]

I'm also Up for this!

[tiffany76]

I'd like to give this one a shot, if that's okay.

As for where to put the new, single document, is the Troubleshooting page the best place? And should we keep the Troubleshooting title or rename the page to Internal Observability...or something else?

[mx-psi]

@tiffany76 All yours :) I would want to wait for a review by @codeboten, but we can start. I was personally thinking about a document separate from the troubleshooting one that talks about observability generally, but I am definitely not a docs expert so if you think a different structure would work better we can also consider it

[tiffany76]

Thanks, @mx-psi! Happy to get any and all reviews.

I was personally thinking about a document separate from the troubleshooting one that talks about observability generally, but I am definitely not a docs expert so if you think a different structure would work better we can also consider it

Okay, makes sense. I'll start pulling the information into a single document while we wait for input from the docs maintainers and others about the best place to put it. And I'll add my own thoughts once I spend some time with the content.

[svrnm]

We can have both a troubleshooting and observability page, they might be linked but overall they have non-overlapping content

[chalin]

There's some prose from a Caution side-note that needs adjustment. It was introduced in #4246, but can be fixed in #4322. Here's the context:

• [ ] https://github.com/open-telemetry/opentelemetry.io/pull/4246/files#r1592148676