New component: Fluent Forward Exporter

[r0mdau]

The purpose and use-cases of the new component

Forward is the protocol used by Fluentd to route message between peers.

The goal is to export logs to Fluentd or Fluentbit process that runs with the in_forward input.

This protocol act on TCP (optional with TLS) using MessagePack object serialization.

• Protocol specification: Forward protocol specification v1
• Library used IBM/fluent-forward-go (MIT License)

Example configuration for the component

fluentforward/allsettings:
  endpoint: "localhost:24224"
  connection_timeout: 30s
  tls:
    insecure: false
    insecure_skip_verify: true
    ca_file: ca.crt
    cert_file: client.crt
    key_file: client.key
  shared_key: "otelcol-dev"
  require_ack: true
  tag: nginx
  compress_gzip: true
  default_labels_enabled:
    time: true
    exporter: false
    job: false
    instance: false
  sending_queue:
    enabled: true
    num_consumers: 2
    queue_size: 10
  retry_on_failure:
    enabled: true
    initial_interval: 10s
    max_interval: 60s
    max_elapsed_time: 10m

Configuration example source: https://github.com/r0mdau/fluentforwardexporter/blob/main/testdata/config.yaml

Telemetry data types supported

Logs

Is this a vendor-specific component?

• [ ] This is a vendor-specific component
• [ ] If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

r0mdau

Sponsor (optional)

No response

Additional context

• The exporter code is currently available here: https://github.com/r0mdau/fluentforwardexporter
• My testing environment to build the exporter available here: https://github.com/r0mdau/otelcol-dev

[r0mdau]

@dmitryax, you may be interested because you own the receiver code.

[bryan-aguilar]

FluentBit already accept OTLP. Why is a forward exporter required?

[r0mdau]

@bryan-aguilar because I bet to not be the only one running infrastructure that only accept fluent-forward protocol as input and the protocol itself has some advantages by operating at OSI layer 4 (TCP) and can use a shared key for authorization. The protocol is using MessagePack objet serialization.

Example:

flowchart LR

A[Opentelemetry-CollectorA] -->|fluent-forward| B(Fluentd)
E[Opentelemetry-CollectorB...] -->|fluent-forward| B
B --> C(OpenSearch)

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[tarokkk]

Also interested in this. A lot of people still use fluentd as an aggregator for specific outputs or doing some specific transformations. However I tried out the plugin and it seems that the receiver and the exporter conflicting

panic: msgp: RegisterExtension() called with typ 0 more than once

goroutine 1 [running]:
github.com/tinylib/msgp/msgp.RegisterExtension(0x0, 0x7a2dc40)
	github.com/tinylib/[email protected]/msgp/extension.go:45 +0x118
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/fluentforwardreceiver.init.0()
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/timeext.go:17 +0x28

[r0mdau]

Thanks for catching the issue @tarokkk , I pushed an update : https://github.com/IBM/fluent-forward-go/pull/63

[OverOrion]

Hey @r0mdau, thanks for this! I gave it a test run with some patches and worked as intended :)

I would like to know if you are planning on opening a PR and upstreaming this?

[r0mdau]

I plan to integrate the update in my repo and yes, if the maintainers accept this issue, I plan to move the code of r0mdau/fluentforwardexporter here.

[r0mdau]

@tarokkk when the exporter is accepted by maintainers, in the MR, will propose to update the msgp.RegisterExtension type from 0 to something like 10 in the fluentforwardreceiver, ref: https://github.com/tinylib/msgp/blob/master/msgp/extension.go#L39

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[github-actions[bot]]

This issue has been closed as inactive because it has been stale for 120 days with no activity.