Segment http - How to add HTTP header

[Nachtfalkeaw]

Hello,

I am evaluating this flowpipeline in my environment. I want to collect netflow from several DataCenter devices like Firewalls, Loadbalancer and Routers. I want to collect these logs using the "goflow" segment and then forarding the results via http to a Grafana Loki environment.

To send data to the Grafana Loki Endpoint I need to configure a header called "X-Scope-OrgID" which identifies to which user these data belongs in a multitenant loki.

Running this simplified config:

- segment: goflow
- segment: http
  config:
    url: https://prometheus.my.domain.de:3100/loki/api/v1/push

results in this error:

[root@u999fmlab001l alloy]# ./flowpipeline-linux-static -c config.yml -l "info"
2025/04/07 11:03:44 [info] Goflow: Scheme sflow supported.
2025/04/07 11:03:44 [info] Goflow: Scheme netflow supported.
2025/04/07 11:03:44 [info] Goflow: Configured for for sflow://:6343,netflow://:2055
2025/04/07 11:03:44 [info] Goflow: 'workers' set to default '1'.
2025/04/07 11:03:44 [info] Goflow: Listening for Netflow v9 on port 2055...
2025/04/07 11:03:44 [info] Goflow: Listening for sflow on port 6343...
2025/04/07 11:03:44 [error] Http: Server endpoint error, skipping at least one flow. Code 401 Unauthorized.
2025/04/07 11:03:44 [error] Above message will not repeat for every flow and is effective until resolved.

Another thing I noticed with this specific error is that it seems to open a single http connection per flow. Is there some possibility to "batch" the flows and send these through one existing connection or define how many parallel connections are allowed ?

[ynHuber]

Hello @Nachtfalkeaw,

this is currently not possible with the very basic implementation of the http segment and would require extending https://github.com/BelWue/flowpipeline/blob/master/segments/alert/http/http.go, which still has many open TODOs and is currently more of a minimum working example.

Theoretically, it would be possible to implement the requirements. Adding the Header-Parameter can simply be done by adding a specific field to the segment and changing the Post Request to set a custom header:

		req, err := http.NewRequest(http.MethodPost, segment.Url, bytes.NewBuffer(data))
		if err != nil {
			log.Error().Err(err).Msg("Http: Request setup error, skipping at least one flow")
		}
		req.Header.Add("ContentType", "application/json")
		req.Header.Add("X-Scope-OrgID", segment.XScopeOrgID)

		resp, err:= http.DefaultClient.Do(req)

Though, a generic solution supporting a custom header field map would be a better way of approaching it. Regarding the batch processing and (re)using parallel requests, I'd have to take a more in-depth look into the implementation of the golang http client options.

[Nachtfalkeaw]

Can be closed If needed. Switched to netobserv flowlogs-pipeline.

Thanks for you time