[docs/collector/scaling, docs/collector/deployment] Describe how to deal with single-writer principle

[mx-psi]

What needs to be changed?

The OpenTelemetry metrics data model requires each metric data stream to have a single writer. This spec section is exhaustive but it is written in spec-speak and it can be a bit hard to understand (and, possibly, to find).

Specifically in the Collector, it can be hard to understand that this is a requirement to avoid overwriting the same time series. Since we already have a couple pages dedicated to how to deploy the Collector, we can explicitly mention taking care of this in the documentation.

What is the name + path of the page that needs changed? Either of collector/scaling or collector/deployment or one of its subpages would be good for this.

Additional context: Example related issue open-telemetry/opentelemetry-collector-contrib/issues/32043

[erharjotsingh]

/assign erharjotsingh Hello @mx-psi Could you please help with some more details on expected results. Thanks

[mx-psi]

@erharjotsingh Sure!

This would be a new section on our documentation that describes, in an user-friendly way:

1. What the single-writer principle is (you can see the link above to the spec definition)
2. What kinds of problems it can cause
3. How you can avoid running into this issue in your Collector deployment.

Additionally, you would have to figure out what page is the best to fit this content: either collector/scaling or collector/deployment seem like a good fit to me.

[svrnm]

@erharjotsingh you still are assigned to this issue, I just removed your assignment since we do not assign issues in our repository to non-regular contributors. Thanks for offering your help! Do not hesitate to raise an incomplete PR with questions for us!

[michael2893]

hello!

I have never contributed before, and I have opened a draft PR.

I really want to make sure I understand the crux of the issue, and I am able to address it in a documentation update. So please do let me know if I am starting in the right place, or if I am a bit off base in what needs discussed.

The change is not ready for serious review, but I want to make sure I am thinking about this correctly as a new contributor.

[svrnm]

Thanks @michael2893, please make sure you sign the CLA, afterwards I am happy to provide you with some initial feedback.

[erharjotsingh]

Thanks @svrnm for offering to look on PR. But as I see now someone else landed and started providing PR, as its unassigned issue.
Hopefully process will be improved in future for this project.

[svrnm]

@erharjotsingh I am very sorry that this happened this way. I didn't realize that between you offering to provide a PR and @michael2893 raising one was only 3 days. This was a human error on my end, which also could not have been avoided by assigning this issue.

[michael2893]

Hi, 👋

I certainly didn't mean to cause an issue here! I'm quite sorry about that.

I think I overlooked the timeframe, and only noted that the issue had been unassigned.

The topic was interesting to me so I think I may have been a bit hasty.

[lahsivjar]

I have a somewhat unique use case where the single writer spec kinda falters a bit. I need to extract some information from traces into metrics and then merge those metrics per collector instance i.e. stripping most of the resource attributes and aggregating them together. The resulting metrics will, of course, not obey the single-writer principle. The way I see it, the writer here is the collector instance since it is producing a new metric so I think the resource attributes should have attributes that uniquely identify a collector instance. What do other members think? Is it a valid use-case?

[svrnm]

@open-telemetry/collector-approvers please take a look at @lahsivjar's question. thanks

[lahsivjar]

The way I see it, the writer here is the collector instance since it is producing a new metric so I think the resource attributes should have attributes that uniquely identify a collector instance

One suggestion/idea I have is that we can use the service.{name, namespace, instance.id} for the running collector instance but under a different prefix - something like observer.service.{name, namespace, instance.id}.