Feature: Generate function and extension logs via Telemetry API receiver

[jerrytfleung]

Motivation

Telemetry API can provide function and extension log. It is better to forward logs so as to provide more observability.

Changes

• Changed Subscribe function so it takes event type slice in telemetryapi/client.go
• Added logs support
• Added sharedcomponent (copied from opentelemetry-collector-contrib) so as to share the http server among receivers
• Added unit test cases for logs

[tylerbenson]

@rapphil @nslaughter PTAL.

[rapphil]

My overall comment is that I'm trying to understand why we need to have this:

A dded sharedcomponent (copied from opentelemetry-collector-contrib) so as to share the http server among receivers

Can't we have multiple subscriptions using the telemetry client, each using a different port to subscribe to the events so that we don't have to have this singleton receiver? This will open the door for adding the configurations that you suggested here, which I think is a good idea.

[jerrytfleung]

My overall comment is that I'm trying to understand why we need to have this:

Added sharedcomponent (copied from opentelemetry-collector-contrib) so as to share the http server among receivers

Can't we have multiple subscriptions using the telemetry client, each using a different port to subscribe to the events so that we don't have to have this singleton receiver? This will open the door for adding the configurations that you suggested here, which I think is a good idea.

I managed to support multiple instances of telemetry api receiver by allowing user to configure port in the telemetry API receiver configuration. Please help to take a look and let me know your feedback. Thanks!

example config

receivers:
    telemetryapi:
    telemetryapi/1:
      port: 4326
    telemetryapi/2:
      port: 4326
      types:
        - platform
        - function
    telemetryapi/3:
      port: 4326
      types: ["platform", "function"]

[jerrytfleung]

@rapphil @nslaughter PTAL. Thanks!

[jerrytfleung]

My overall comment is that I'm trying to understand why we need to have this:

A dded sharedcomponent (copied from opentelemetry-collector-contrib) so as to share the http server among receivers

Can't we have multiple subscriptions using the telemetry client, each using a different port to subscribe to the events so that we don't have to have this singleton receiver? This will open the door for adding the configurations that you suggested here, which I think is a good idea.

I am thinking the following configuration so users can configure the types of subscription per each receivers. Also they can put the receiver to the pipeline according to their requirements.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "localhost:4317"
      http:
        endpoint: "localhost:4318"
  telemetryapi/1:
    port: 4325
    types: [platform]
  telemetryapi/2:
    port: 4326
    types: [function]
exporters:
  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp,telemetryapi/1,telemetryapi/2]
      exporters: [logging]
    metrics:
      receivers: [otlp]
      exporters: [logging]
    logs/1:
      receivers: [telemetryapi/1]
      exporters: [logging]
    logs/2:
      receivers: [telemetryapi/2]
      exporters: [logging]
  telemetry:
    metrics:
      address: localhost:8888

[bhaskarbanerjee]

@jerrytfleung I see that the config you shared above is supporting metric in the pipeline as well? metrics:

metrics:
      receivers: [otlp]
      exporters: [logging]

But as far I see this I can only see it supporting traces.

Do you mean that with this PR, the collector will start receiving and exporting metric as well as logs, in addition to the traces?

[jerrytfleung]

@jerrytfleung I see that the config you shared above is supporting metric in the pipeline as well? metrics:

metrics:
      receivers: [otlp]
      exporters: [logging]

But as far I see this I can only see it supporting traces.

Do you mean that with this PR, the collector will start receiving and exporting metric as well as logs, in addition to the traces?

@bhaskarbanerjee Solarwinds has plan to extract information from platform event and convert them into metrics. e.g. When we see phase=init from initReport, we can increase the counter of fass.coldstarts and it is a FaaS metrics. Currently, it is still WIP from Solarwinds due to some internal discussion.

This PR covers logs only.

[tylerbenson]

@rapphil @jerrytfleung What's the status of this PR? Are there more changes needed, or is resolving the merge conflict enough?

@jerrytfleung maybe you could come to the next SIG meeting and present about these changes? I'm having a hard time understanding and I think that would help.

[jerrytfleung]

@rapphil @jerrytfleung What's the status of this PR? Are there more changes needed, or is resolving the merge conflict enough?

@jerrytfleung maybe you could come to the next SIG meeting and present about these changes? I'm having a hard time understanding and I think that would help.

I resolved the merge conflict. I don't have other changes to add. This PR targets to extract the log messages in telemetry API and export them.

I am happy to join the SIG meeting to present the idea / result and answer questions. Will see you then.

[jerrytfleung]

@tylerbenson @nslaughter I updated the test cases and polished the code again. Please help to proceed. Thanks!