Support textPayload for Stackdriver Output Plugin

[HarleyB123]

When using the Stackdriver plugin, I notice that all logs appear under the format:

jsonPayload {
    "log":"blah",
    ...
}

I would like for textPayload to be supported, such that I can have an output like:

textPayload: "blah"

without any jsonPayload field.

It may be that this is already supported, as the managed fluentbit-gke daemonset for GCP seems to display both textPayload and jsonPayload fields, but I'm unsure whether this is due to fluentbit or the fluent-bit-gke-exporter image. If this is supported, instructions on how to get the textPayload field would be appreciated.

[BartVB]

I'm in the same boat. Trying to get a k8s cluster to send all logs to Google Cloud Logging but that has proven to be quite a challenge if you want more than just time + log message.

GKE is using Fluent bit for log collection so I also looked at that for some inspiration but it seems like gke.gcr.io/fluent-bit:v1.5.7-gke.2 isn't using the stackdriver output nor the kubernetes filter. Fluent-bit is sending all logs to port 2021 which is gke.gcr.io/fluent-bit-gke-exporter:v0.17.1-gke.0 which mainly contains one fluent-bit-gke-exporter Go-binary without any documentation or configuration. This binary seems to be called gke-logmon during development.

Getting Fluent Bit to work properly (i.e. including filters) seems to be very non trivial, or I'm missing something in a big way.

@HarleyB123 Why do you want to move this data? You could use a filter to rename it to message or move/copy it and use the nest filter to move it up in the JSON structure. https://cloud.google.com/logging/docs/structured-logging

[HarleyB123]

You could use a filter to rename it to message or move/copy it and use the nest filter to move it up in the JSON structure.

Renaming to message didn't work in my case, I just got:

jsonPayload {
    "message":"blah",
    ...
}

When trying to lift it out of the jsonPayload field, the following appears to have no effect:

[FILTER]
    Name nest
    Match *
    Operation lift
    Nested_under jsonPayload

[ak185158]

We're running into the same challenge in trying to forward logs to cloud logging. For certain logs, we want the message to be textPayload in order to match the format that is seen in GKE managed logs.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[BartVB]

As far as I know this is still an issue with the latest FluentBit version.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[BartVB]

A fix (or even a workaround) would still be appreciated.

[leonardo-albertovich]

In order to do that we would need to make the following changes:

1. Add these constants * here :
2. Add a new flb_stackdriver field here for the textual payload type and one for a numerical id so the flush process is simpler
3. Add a new config map string entry here for the payload type (I'd suggest naming it payload_type) which defaults to FLB_STD_PAYLOAD_TYPE_JSON
4. Add a conditional here based on the value stored in ctx->payload_type_id and then you can wrap whe whole 2211 - 2222 block in the json side of the conditional and add the plaintext counterpart which means you will have to create a function named pack_text_payload which should be an extremely simplified version of the same function that just packs the log line as a string instead.
5. Add some code here to select the right value for ctx->payload_type_id based on what came in ctx->payload_type (the string we get from the user) but be sure to default to FLB_STD_PAYLOAD_TYPE_JSON_ID because we can't silently change the default behavior, that would make a lot of people angry.

Constants :

FLB_STD_PAYLOAD_TYPE_JSON = "json"
FLB_STD_PAYLOAD_TYPE_TEXT = "text"

FLB_STD_PAYLOAD_TYPE_JSON_ID = 0
FLB_STD_PAYLOAD_TYPE_TEXT_ID = 1

I can't assure you that such a PR would get merged since that's not my decision but if anyone decides to work on it feel free to ping me in slack if you have any doubts or get stuck and I will help you.

Regards.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.