Allow templating headers in `http` sink

[binarylogic]

For certain services the ability to dynamically set HTTP headers based on log content is important. From a user on a related project:

I am sending Docker container logs to Sumo Logic, and would like to dynamically set log metadata based on the incoming log records. Sumo Logic's HTTP receivers accept custom source names, hosts, and categories by setting HTTP headers named X-Sumo-Name, X-Sumo-Host, X-Sumo-Category.

Describe the solution you'd like

I would like a way to set dynamically header values in the http output plugin. For example, let's say there is a record key named category. There should be a way to use this as the header value.

[1154103724, {"key1"=>12345, "category"=>"abc"}]

[sinks.sumologic]
    Name           http
    Match          *
    Host           127.0.0.1
    Port           9000
    Header         X-Key-A Value_A
    Header_record  X-Key-B category
    URI            /something

In the above configuration, the header X-Key-A would have the value Value_A, but X-Key-B would have the value abc from the record.

Additional context

Without the ability to customize the log metadata via headers, I cannot effectively sort and manage logs in Sumo Logic.

[binarylogic]

Thinking about this more, how do we handle multiple different values? Does it create multiple buffers / requests? In a way, I'm thinking this should be addressed with a higher-level feature like a "group_by" transform.

[jalaziz]

Still new to Rust, but could this be solved with the relatively new PartitionHttpSink?

[binarylogic]

cc @bruceg on that one.

[bruceg]

Yes, PartitionHttpSink looks to be the ideal target for this kind of modification, and would end up being ideal sample code for other HTTP sinks to adapt when they need this kind of enhancement. The key to the partition types would be the HTTP headers so that the sink batches events with like headers together.

[jszwedko]

Closed by https://github.com/vectordotdev/vector/issues/7189

[jalaziz]

@jszwedko Has this actually been fixed? #7189 was an issue that was closed in favor of this one. I can't seem to find a PR related to either.

[jszwedko]

Hi @jalaziz !

Apologies, you are right, I misread this as just allowing headers to be set, but I see this is about allowing templated HTTP headers. I'll reopen.

[goedelsoup-clari]

The original request is specifically to support the use of dynamic source categories within SumoLogic. Is templating support through the use of PartitionHttpSink still preferred or would a SumoLogic logs sink be another option at this point?

We are beginning to launch a Vector-first implementation of our observability topology on EKS and while we can use the current HTTP sink in conjunction with Field Extraction Rules to make things palatable, it becomes hard to manage lifecycles of logs and it doesn't integrate well with enabling development teams to manage pipelines and data stores/retention independently.

Having the full control of a dedicated sink would be great, but templating via headers does directly solve the problem. Would be happy to try to get company approval to tackle this if the Vector team is in agreement on the approach.

[jszwedko]

Hi @goedelsoup-clari !

Regarding having http headers be templatable we are definitely open to this. It's a fairly common request even outside of the context of sending logs to SumoLogic. It would mean partitioning the batches using the header values.

We'd very likely be open to a dedicated SumoLogic sink too. If you want to pursue that route, would you want to open a separate issue? We have this checklist that we like to run through for new component proposals: https://github.com/vectordotdev/vector/blob/master/CONTRIBUTING.md#new-sources-sinks-and-transforms

[DGolubets]

This feature would be nice to have.

[sreisich]

I need exactly the same for Parseable. In order for a stream to be created in Parseable, I need to set the header X-P-Stream. And I need the streams to be the kubernetes namespaces:

request:
    headers:
        X-P-Stream: "{{ .kubernetes_pod_namespace }}"

Any chance this will be implemented soon?

[matiaskotlik]

I also need templating in the uri field for my use-case.

[a-abella]

I was all excited to use header templating to make Vector function as a Mimir-tenant router with a disk buffer, I thought it would be possible since the prometheus_remote_write sink has templated headers,

  schema:
    log_namespace: true
  
  sources:
    # use http_server sources for all data types, to route on headers
    # https://vector.dev/docs/reference/configuration/sources/http_server/
    src_mimir:
      type: http_server
      address: 0.0.0.0:9090
      encoding: binary
      headers:
        - X-Scope-OrgID
      
  sinks:
    dst_mimir:
      inputs: [src_mimir]
      request:
        headers:
          Content-Type: application/x-protobuf
          X-Scope-OrgID: |
            {{ print "{{ %http_server.headers.\"X-Scope-OrgID\" }}" }}
      type: http
      uri: "http://192.168.1.5:9009/api/v1/push"
      compression: snappy
      encoding:
        codec: raw_message
      framing:
        method: bytes

This would be awesome, and so much better than using transform routes -- and having to update the routes and sinks for every new tenant.

[solidDoWant]

This would be really nice to have with the elasticsearch provider as well. VictoriaLogs, which uses an elasticsearch-compatible API for ingestion, needs the VL-Stream-Fields header or _stream_fields query parameter set for performance reasons. This header is heavily dependent on the source of the logs, so for example, journald and kubernetes log sources should set a different value for this header. Without the ability to template either headers or query parameters, every single source type used needs a corresponding sink.

[vinzee]

Hi folks, just following up on this issue/request.

We need templatized headers/params for writing to databricks or managed-iceberg-tables (like aws s3tables).

The clickhouse sink already supports templatized params, so i'm curious if support for this can be added in the generic http sink as well.

Thank you!

[pront]

We are reviewing https://github.com/vectordotdev/vector/pull/23422. It should be part of the next Vector release.

[mattnthat]

I also need templating in the uri field for my use-case.

@pront Im guessing this requirement would need to fall into a separate issue / work item?

[pront]

I also need templating in the uri field for my use-case.

@pront Im guessing this requirement would need to fall into a separate issue / work item?

Yes, I recommend creating a new feature request.