Json parser can not apply `null_value_pattern` to nested values

[andrew-pickin-epi]

Describe the bug

Json parser not respecting null_value_pattern for nested values

To Reproduce

https://github.com/fluent/fluentd/issues/4222#issuecomment-1627874181

old

{msg: "foo", "ConfigMap/logging/fluentd-config":"unchanged","ConfigMap/logging/index-template":"unchanged"}

null_value_pattern unchanged should produce:

{msg: "foo"}

Expected behavior

Field with value unchanged should be discarded.

Your Environment

- Fluentd version: v1.15.3-debian
- TD Agent version: n/a
- Operating system:
- Kernel version: AWS EKS

Your Configuration

<filter kubernetes.log.flux-system.**>
  @id parser_flux
  @type parser
  key_name "log"
  reserve_data true
  remove_key_name_field true
  inject_key_prefix "log_"
  emit_invalid_record_to_error false
  <parse>
    @type "json"
    null_value_pattern unchanged
  </parse>
</filter>

### Your Error Log

```shell
No errors

Additional context

I'm trying to discard a large number of fields (list is dynamic) with anything of value unchanged. alternative approaches welcome.

[daipom]

null_value_pattern unchanged should produce:

{msg: "foo"}

Expected behavior

Field with value unchanged should be discarded.

No. The expected result is as follows.

{"msg":"foo","ConfigMap/logging/fluentd-config":null,"ConfigMap/logging/index-template":null}

We use null_value_pattern to recognize some patterns as NULL value. It is not to discard keys.

[andrew-pickin-epi]

{"msg":"foo","ConfigMap/logging/fluentd-config":null,"ConfigMap/logging/index-template":null}

Ok, this isn't my desired result, but this doesn't happen either.

[daipom]

What happens actually? Please share the actual result.

[andrew-pickin-epi]

Sorry, hadn't realised I hadn't posted that. Output is unchanged (literally):

{msg: "foo", "ConfigMap/logging/fluentd-config":"unchanged","ConfigMap/logging/index-template":"unchanged"}

As I said, what I'd like is {msg: "foo}" and suggestions as to how this can be achieved would be very welcome.

[andrew-pickin-epi]

Actually, this needs more qualification. input

{"error":null,"level":"info","msg":"Main container completed","time":"2023-06-30T16:43:53.152Z"}

and config

  @type parser
  key_name "log"
  reserve_time true
  reserve_data true
  remove_key_name_field true
  inject_key_prefix "log_"
  emit_invalid_record_to_error false
  <parse>
    @type "json"
  </parse>

Gives:

key	value
log_level	info
log_msg	Main container completed

Note: no log_error, so a null value doesn't create a field.

So {"msg":"foo","ConfigMap/logging/fluentd-config":null,"ConfigMap/logging/index-template":null}

With the above config with null_value_pattern unchanged, would expect log_msg only.

[daipom]

Note: no log_error, so a null value doesn't create a field.

I think it is not the specification of Parser plugin. Some other settings may discard the key o the value null, such as Formatter of Output plugin.

I can't reproduce it.

config:

<source>
  @type tail
  tag test
  path /path/to/test.log
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<filter test.**>
  @type parser
  key_name message
  reserve_data true
  remove_key_name_field true
  inject_key_prefix log_
  emit_invalid_record_to_error false
  <parse>
    @type json
  </parse>
</filter>

<match test.**>
  @type stdout
</match>

/path/to/test.log:

{"error":null,"level":"info","msg":"Main container completed","time":"2023-06-30T16:43:53.152Z"}

Result:

2023-07-01 11:26:45 +0900 [info]: #0 starting fluentd worker pid=32125 ppid=32121 worker=0
2023-07-01 11:26:45 +0900 [info]: #0 following tail of /path/to/test.log
1970-01-01 09:33:43.000000152 +0900 test: {"log_error":null,"log_level":"info","log_msg":"Main container completed"}
2023-07-01 11:26:45 +0900 [info]: #0 fluentd worker is now running worker=0

[daipom]

In addition, the following setting with null_value_pattern works as expected.

<source>
  @type tail
  tag test
  path /path/to/test.log
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<filter test.**>
  @type parser
  key_name message
  reserve_data true
  remove_key_name_field true
  inject_key_prefix log_
  emit_invalid_record_to_error false
  <parse>
    @type json
    null_value_pattern unchanged
  </parse>
</filter>

<match test.**>
  @type stdout
</match>

/path/to/test.log:

{"error":null,"level":"info","msg":"unchanged","time":"2023-06-30T16:43:53.152Z"}

Result:

2023-07-01 11:35:20 +0900 [info]: #0 starting fluentd worker pid=32806 ppid=32802 worker=0
2023-07-01 11:35:20 +0900 [info]: #0 following tail of /path/to/test.log
1970-01-01 09:33:43.000000152 +0900 test: {"log_error":null,"log_level":"info","log_msg":null}
2023-07-01 11:35:20 +0900 [info]: #0 fluentd worker is now running worker=0

[daipom]

Please check this behavior in your environment, and please find out what is causing that behavior. Since you are able to discard nulls (I'm not sure which setting is working though...), perhaps you can fix some configs and it will work.

If you find other settings that seem to be the cause, you can ask us again! I don't think this is a bug, so perhaps you should ask your question here:

• https://github.com/fluent/fluentd/discussions/categories/q-a

[andrew-pickin-epi]

Your example isn't quite the same, I have multiple instances I want to be null and more complex field names. Irrespective of the format output, the json is being being parsed and yet I have multiple field that aren't being nulled .

[daipom]

I just wanted to say I think this is not a bug of Parser, and you can confirm it by this example.

[andrew-pickin-epi]

I understand it works in your example, but the same config doesn't work consistently. I'm suggesting your example is too simplistic. My config:

  <parse>
    @type "json"
    null_value_pattern unchanged
  </parse>

Still results in fields with values unchanged:

{"log_msg":"foo","log_ConfigMap/logging/fluentd-config":"unchanged","log_ConfigMap/logging/index-template":"unchanged"}

Even thought it's going through the json parser.

[daipom]

Could you summarize how to reproduce that in this way: https://github.com/fluent/fluentd/issues/4222#issuecomment-1615381247? It is important to summarize and share the general reproduction way.

[andrew-pickin-epi]

This is my real-world example.

Log from pod, specifically fluxcd/flux kustomize controller:

{"level":"info","ts":"2023-06-29T14:48:48.229Z","msg":"server-side apply completed","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"eks-cluster-konf","namespace":"flux-system"},"namespace":"flux-system","name":"eks-cluster-konf","reconcileID":"06ad0250-3519-4e0a-8889-ffe3d48dd2cf","output":{"ConfigMap/cbd-production/cbd-api-k6bk5m8d82":"unchanged","ConfigMap/cbd-production/cbd-api-logback-config-h97k79tk9d":"unchanged","ConfigMap/cbd-production/cbd-app-hd2b762mk2":"unchanged","ConfigMap/cbd-production/cbd-podium-config-mt9m5dhkk7":"unchanged","ConfigMap/cbd-production/cbd-podium-fuseki-config":"unchanged","ConfigMap/cbd-production/cbd-podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/cbd-production/cbd-podium-redis-config":"unchanged","ConfigMap/cbd-production/data-consumer-config-td9bm259b5":"unchanged","ConfigMap/cbd-testing/cbd-api-k6bk5m8d82":"unchanged","ConfigMap/cbd-testing/cbd-api-logback-config-h97k79tk9d":"unchanged","ConfigMap/cbd-testing/cbd-app-hd2b762mk2":"unchanged","ConfigMap/cbd-testing/cbd-podium-config-d625d942gc":"unchanged","ConfigMap/cbd-testing/cbd-podium-fuseki-config":"unchanged","ConfigMap/cbd-testing/cbd-podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/cbd-testing/cbd-podium-redis-config":"unchanged","ConfigMap/cbd-testing/data-consumer-config-48m8dct24b":"unchanged","ConfigMap/registry-dev/build-script":"unchanged","ConfigMap/registry-dev/registry-config":"unchanged","ConfigMap/registry-dev/registry-scripts":"unchanged","ConfigMap/registry-dev/tdb-config":"unchanged","ConfigMap/registry-prod/build-script":"unchanged","ConfigMap/registry-prod/registry-config":"unchanged","ConfigMap/registry-prod/registry-scripts":"unchanged","ConfigMap/registry-prod/tdb-config":"unchanged","ConfigMap/registry-staging/build-script":"unchanged","ConfigMap/registry-staging/registry-config":"unchanged","ConfigMap/registry-staging/registry-scripts":"unchanged","ConfigMap/registry-staging/tdb-config":"unchanged","ConfigMap/rp-production/data-consumer-config-bd9tft494k":"unchanged","ConfigMap/rp-production/rp-api-996f76g9m9":"unchanged","ConfigMap/rp-production/rp-api-logback-config-h97k79tk9d":"unchanged","ConfigMap/rp-production/rp-app-8mgt7mdkg4":"unchanged","ConfigMap/rp-production/rp-podium-config-t5457b6kdk":"unchanged","ConfigMap/rp-production/rp-podium-fuseki-config":"unchanged","ConfigMap/rp-production/rp-podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/rp-production/rp-podium-redis-config":"unchanged","ConfigMap/rp-testing/data-consumer-config-k2h5hbtf92":"unchanged","ConfigMap/rp-testing/rp-api-996f76g9m9":"unchanged","ConfigMap/rp-testing/rp-api-logback-config-h97k79tk9d":"unchanged","ConfigMap/rp-testing/rp-app-8mgt7mdkg4":"unchanged","ConfigMap/rp-testing/rp-podium-config-844888k8tt":"unchanged","ConfigMap/rp-testing/rp-podium-fuseki-config":"unchanged","ConfigMap/rp-testing/rp-podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/rp-testing/rp-podium-redis-config":"unchanged","ConfigMap/rp-uat/data-consumer-config-7dbm4dt49f":"unchanged","ConfigMap/rp-uat/rp-api-996f76g9m9":"unchanged","ConfigMap/rp-uat/rp-api-logback-config-h97k79tk9d":"unchanged","ConfigMap/rp-uat/rp-app-8mgt7mdkg4":"unchanged","ConfigMap/rp-uat/rp-podium-config-69kh6f6749":"unchanged","ConfigMap/rp-uat/rp-podium-fuseki-config":"unchanged","ConfigMap/rp-uat/rp-podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/rp-uat/rp-podium-redis-config":"unchanged","ConfigMap/rpa-dev/data-consumer-config-4f688ht5t2":"unchanged","ConfigMap/rpa-dev/rpa-api-dcgb7khchd":"unchanged","ConfigMap/rpa-dev/rpa-api-logback-config-h97k79tk9d":"unchanged","ConfigMap/rpa-dev/rpa-app-tm4b7dbk2g":"unchanged","ConfigMap/rpa-dev/rpa-podium-config-g994f8242c":"unchanged","ConfigMap/rpa-dev/rpa-podium-fuseki-config":"unchanged","ConfigMap/rpa-dev/rpa-podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/rpa-dev/rpa-podium-redis-config":"unchanged","ConfigMap/uv-dev/build-script":"unchanged","ConfigMap/uv-dev/consumer-config-f56tf4mt28":"unchanged","ConfigMap/uv-dev/podium-config-fhhggtg6fc":"unchanged","ConfigMap/uv-dev/podium-fuseki-config":"unchanged","ConfigMap/uv-dev/podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/uv-dev/podium-redis-config":"unchanged","ConfigMap/uv-dev/tdb-config":"unchanged","ConfigMap/uv-dev/tdb-stats":"unchanged","ConfigMap/uv-dev/uv-api-mg4dbb957t":"unchanged","ConfigMap/uv-dev/uv-context-file-dtc67ck577":"unchanged","CronJob/uv-dev/podium-backup":"unchanged","CronJob/uv-dev/podium-redis-pruning":"unchanged","CronWorkflow/registry-staging/registry-image-build":"unchanged","Deployment/cbd-production/cbd-api":"unchanged","Deployment/cbd-production/cbd-app":"unchanged","Deployment/cbd-production/cbd-podium-manager":"unchanged","Deployment/cbd-production/cbd-podium-publisher":"unchanged","Deployment/cbd-production/fuseki-consumer":"unchanged","Deployment/cbd-testing/cbd-api":"unchanged","Deployment/cbd-testing/cbd-app":"unchanged","Deployment/cbd-testing/cbd-podium-manager":"unchanged","Deployment/cbd-testing/cbd-podium-publisher":"unchanged","Deployment/cbd-testing/fuseki-consumer":"unchanged","Deployment/rp-production/fuseki-consumer":"unchanged","Deployment/rp-production/rp-api":"unchanged","Deployment/rp-production/rp-app":"unchanged","Deployment/rp-production/rp-podium-manager":"unchanged","Deployment/rp-production/rp-podium-publisher":"unchanged","Deployment/rp-testing/fuseki-consumer":"unchanged","Deployment/rp-testing/rp-api":"unchanged","Deployment/rp-testing/rp-app":"unchanged","Deployment/rp-testing/rp-podium-manager":"unchanged","Deployment/rp-testing/rp-podium-publisher":"unchanged","Deployment/rp-uat/fuseki-consumer":"unchanged","Deployment/rp-uat/rp-api":"unchanged","Deployment/rp-uat/rp-app":"unchanged","Deployment/rp-uat/rp-podium-manager":"unchanged","Deployment/rp-uat/rp-podium-publisher":"unchanged","Deployment/rpa-dev/fuseki-consumer":"unchanged","Deployment/rpa-dev/rpa-api":"unchanged","Deployment/rpa-dev/rpa-app":"unchanged","Deployment/rpa-dev/rpa-podium-manager":"unchanged","Deployment/rpa-dev/rpa-podium-publisher":"unchanged","Deployment/uv-dev/fuseki-json-consumer":"unchanged","Deployment/uv-dev/podium-manager":"unchanged","Deployment/uv-dev/podium-publisher":"unchanged","Deployment/uv-dev/uv-api":"unchanged","ImagePolicy/cbd-production/cbd-api":"unchanged","ImagePolicy/cbd-production/cbd-app":"unchanged","ImagePolicy/cbd-testing/cbd-api":"unchanged","ImagePolicy/cbd-testing/cbd-app":"unchanged","ImagePolicy/flux-system/orca-fuseki-consumer":"unchanged","ImagePolicy/registry-dev/fsa-registry-config":"unchanged","ImagePolicy/registry-prod/fsa-registry-config":"unchanged","ImagePolicy/registry-staging/fsa-registry-config":"unchanged","ImagePolicy/rp-production/rp-api":"unchanged","ImagePolicy/rp-production/rp-app":"unchanged","ImagePolicy/rp-testing/rp-api":"unchanged","ImagePolicy/rp-testing/rp-app":"unchanged","ImagePolicy/rp-uat/rp-api":"unchanged","ImagePolicy/rp-uat/rp-app":"unchanged","ImagePolicy/rpa-dev/rpa-api":"unchanged","ImagePolicy/rpa-dev/rpa-app":"unchanged","ImagePolicy/uv-dev/fuseki-json-consumer":"unchanged","ImagePolicy/uv-dev/uv-api":"unchanged","ImageRepository/cbd-production/cbd-api":"unchanged","ImageRepository/cbd-production/cbd-app":"unchanged","ImageRepository/cbd-testing/cbd-api":"unchanged","ImageRepository/cbd-testing/cbd-app":"unchanged","ImageRepository/flux-system/orca-fuseki-consumer":"unchanged","ImageRepository/registry-dev/fsa-registry-config":"unchanged","ImageRepository/registry-prod/fsa-registry-config":"unchanged","ImageRepository/registry-staging/fsa-registry-config":"unchanged","ImageRepository/rp-production/rp-api":"unchanged","ImageRepository/rp-production/rp-app":"unchanged","ImageRepository/rp-testing/rp-api":"unchanged","ImageRepository/rp-testing/rp-app":"unchanged","ImageRepository/rp-uat/rp-api":"unchanged","ImageRepository/rp-uat/rp-app":"unchanged","ImageRepository/rpa-dev/rpa-api":"unchanged","ImageRepository/rpa-dev/rpa-app":"unchanged","ImageRepository/uv-dev/fuseki-json-consumer":"unchanged","ImageRepository/uv-dev/uv-api":"unchanged","ImageUpdateAutomation/cbd-production/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/cbd-testing/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/registry-dev/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/registry-prod/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/registry-staging/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/rp-production/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/rp-testing/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/rp-uat/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/rpa-dev/fsa-eks-cluster":"unchanged","ImageUpdateAutomation/uv-dev/fsa-eks-cluster":"unchanged","Ingress/cbd-production/cbd-api":"unchanged","Ingress/cbd-production/cbd-app":"unchanged","Ingress/cbd-testing/cbd-api":"unchanged","Ingress/cbd-testing/cbd-app":"unchanged","Ingress/registry-dev/registry":"unchanged","Ingress/registry-prod/registry":"unchanged","Ingress/registry-staging/registry":"unchanged","Ingress/rn-prod/rn":"unchanged","Ingress/rp-production/rp-api":"unchanged","Ingress/rp-production/rp-app":"unchanged","Ingress/rp-testing/rp-api":"unchanged","Ingress/rp-testing/rp-app":"unchanged","Ingress/rp-uat/rp-api":"unchanged","Ingress/rp-uat/rp-app":"unchanged","Ingress/rpa-dev/rpa-api":"unchanged","Ingress/rpa-dev/rpa-app":"unchanged","Ingress/uv-dev/uv-api":"unchanged","PersistentVolume/efs-pv-registry-dev":"unchanged","PersistentVolume/efs-pv-uv-dev":"unchanged","PersistentVolumeClaim/registry-dev/efs-claim":"unchanged","PersistentVolumeClaim/uv-dev/efs-claim":"unchanged","PodDisruptionBudget/cbd-production/cbd-api-pdb":"unchanged","PodDisruptionBudget/cbd-production/cbd-app-pdb":"unchanged","PodDisruptionBudget/cbd-production/cbd-podium-worker-pdb":"unchanged","PodDisruptionBudget/registry-prod/registry-pdb":"unchanged","PodDisruptionBudget/rn-prod/rn-pdb":"unchanged","PodDisruptionBudget/rp-production/rp-api-pdb":"unchanged","PodDisruptionBudget/rp-production/rp-app-pdb":"unchanged","PodDisruptionBudget/rp-production/rp-podium-worker-pdb":"unchanged","Role/registry-prod/release-role":"unchanged","RoleBinding/registry-prod/release-rolebinding":"unchanged","SealedSecret/uv-dev/batch-api-secrets":"unchanged","Service/cbd-production/cbd-api":"unchanged","Service/cbd-production/cbd-app":"unchanged","Service/cbd-production/cbd-podium-manager":"unchanged","Service/cbd-production/cbd-podium-publisher":"unchanged","Service/cbd-production/cbd-podium-redis":"unchanged","Service/cbd-production/cbd-podium-worker":"unchanged","Service/cbd-testing/cbd-api":"unchanged","Service/cbd-testing/cbd-app":"unchanged","Service/cbd-testing/cbd-podium-manager":"unchanged","Service/cbd-testing/cbd-podium-publisher":"unchanged","Service/cbd-testing/cbd-podium-redis":"unchanged","Service/cbd-testing/cbd-podium-worker":"unchanged","Service/registry-dev/registry":"unchanged","Service/registry-prod/registry":"unchanged","Service/registry-staging/registry":"unchanged","Service/rn-prod/rn":"unchanged","Service/rp-production/rp-api":"unchanged","Service/rp-production/rp-app":"unchanged","Service/rp-production/rp-podium-manager":"unchanged","Service/rp-production/rp-podium-publisher":"unchanged","Service/rp-production/rp-podium-redis":"unchanged","Service/rp-production/rp-podium-worker":"unchanged","Service/rp-testing/rp-api":"unchanged","Service/rp-testing/rp-app":"unchanged","Service/rp-testing/rp-podium-manager":"unchanged","Service/rp-testing/rp-podium-publisher":"unchanged","Service/rp-testing/rp-podium-redis":"unchanged","Service/rp-testing/rp-podium-worker":"unchanged","Service/rp-uat/rp-api":"unchanged","Service/rp-uat/rp-app":"unchanged","Service/rp-uat/rp-podium-manager":"unchanged","Service/rp-uat/rp-podium-publisher":"unchanged","Service/rp-uat/rp-podium-redis":"unchanged","Service/rp-uat/rp-podium-worker":"unchanged","Service/rpa-dev/rpa-api":"unchanged","Service/rpa-dev/rpa-app":"unchanged","Service/rpa-dev/rpa-podium-manager":"unchanged","Service/rpa-dev/rpa-podium-publisher":"unchanged","Service/rpa-dev/rpa-podium-redis":"unchanged","Service/rpa-dev/rpa-podium-worker":"unchanged","Service/uv-dev/fuseki-json-consumer":"unchanged","Service/uv-dev/podium-manager":"unchanged","Service/uv-dev/podium-publisher":"unchanged","Service/uv-dev/podium-redis":"unchanged","Service/uv-dev/podium-worker":"unchanged","Service/uv-dev/uv-api":"unchanged","Service/uv-dev/uv-rds":"unchanged","ServiceMonitor/cbd-production/cbd-api":"unchanged","ServiceMonitor/cbd-production/cbd-app":"unchanged","ServiceMonitor/cbd-production/cbd-podium-manager":"unchanged","ServiceMonitor/cbd-production/cbd-podium-publisher":"unchanged","ServiceMonitor/cbd-production/cbd-podium-worker":"unchanged","ServiceMonitor/cbd-testing/cbd-api":"unchanged","ServiceMonitor/cbd-testing/cbd-app":"unchanged","ServiceMonitor/cbd-testing/cbd-podium-manager":"unchanged","ServiceMonitor/cbd-testing/cbd-podium-publisher":"unchanged","ServiceMonitor/cbd-testing/cbd-podium-worker":"unchanged","ServiceMonitor/rn-prod/rn":"unchanged","ServiceMonitor/rp-production/rp-api":"unchanged","ServiceMonitor/rp-production/rp-app":"unchanged","ServiceMonitor/rp-production/rp-podium-manager":"unchanged","ServiceMonitor/rp-production/rp-podium-publisher":"unchanged","ServiceMonitor/rp-production/rp-podium-worker":"unchanged","ServiceMonitor/rp-testing/rp-api":"unchanged","ServiceMonitor/rp-testing/rp-app":"unchanged","ServiceMonitor/rp-testing/rp-podium-manager":"unchanged","ServiceMonitor/rp-testing/rp-podium-publisher":"unchanged","ServiceMonitor/rp-testing/rp-podium-worker":"unchanged","ServiceMonitor/rp-uat/rp-api":"unchanged","ServiceMonitor/rp-uat/rp-app":"unchanged","ServiceMonitor/rp-uat/rp-podium-manager":"unchanged","ServiceMonitor/rp-uat/rp-podium-publisher":"unchanged","ServiceMonitor/rp-uat/rp-podium-worker":"unchanged","ServiceMonitor/rpa-dev/rpa-api":"unchanged","ServiceMonitor/rpa-dev/rpa-app":"unchanged","ServiceMonitor/rpa-dev/rpa-podium-manager":"unchanged","ServiceMonitor/rpa-dev/rpa-podium-publisher":"unchanged","ServiceMonitor/rpa-dev/rpa-podium-worker":"unchanged","ServiceMonitor/uv-dev/fuseki-json-consumer":"unchanged","ServiceMonitor/uv-dev/podium-manager":"unchanged","ServiceMonitor/uv-dev/podium-publisher":"unchanged","ServiceMonitor/uv-dev/podium-worker":"unchanged","ServiceMonitor/uv-dev/uv-api":"unchanged","StatefulSet/cbd-production/cbd-podium-redis":"unchanged","StatefulSet/cbd-production/cbd-podium-worker":"unchanged","StatefulSet/cbd-testing/cbd-podium-redis":"unchanged","StatefulSet/cbd-testing/cbd-podium-worker":"unchanged","StatefulSet/registry-dev/registry":"unchanged","StatefulSet/registry-prod/registry":"unchanged","StatefulSet/registry-staging/registry":"unchanged","StatefulSet/rn-prod/rn":"unchanged","StatefulSet/rp-production/rp-podium-redis":"unchanged","StatefulSet/rp-production/rp-podium-worker":"unchanged","StatefulSet/rp-testing/rp-podium-redis":"unchanged","StatefulSet/rp-testing/rp-podium-worker":"unchanged","StatefulSet/rp-uat/rp-podium-redis":"unchanged","StatefulSet/rp-uat/rp-podium-worker":"unchanged","StatefulSet/rpa-dev/rpa-podium-redis":"unchanged","StatefulSet/rpa-dev/rpa-podium-worker":"unchanged","StatefulSet/uv-dev/podium-redis":"unchanged","StatefulSet/uv-dev/podium-worker":"unchanged","WorkflowTemplate/registry-dev/publish-prep":"unchanged","WorkflowTemplate/registry-dev/registry-image-build":"unchanged","WorkflowTemplate/registry-dev/registry-restart":"unchanged","WorkflowTemplate/registry-prod/publish-prep":"unchanged","WorkflowTemplate/registry-prod/registry-image-build":"unchanged","WorkflowTemplate/registry-prod/registry-restart":"unchanged","WorkflowTemplate/registry-staging/publish-prep":"unchanged","WorkflowTemplate/registry-staging/registry-image-build":"unchanged","WorkflowTemplate/registry-staging/registry-restart":"unchanged"},"revision":"master/a41efdbc4a08041e2ffc0bd092138163477e8a95"}

The relevant config

<filter kubernetes.log.json.**>
  @id raw_json
  @type record_transformer
  <record>
    raw ${record["log"]}
  </record>
</filter>

<filter kubernetes.log.json.**>
  @id parser_argo
  @type parser
  key_name "log"
  reserve_time true
  reserve_data true
  remove_key_name_field true
  inject_key_prefix "log_"
  emit_invalid_record_to_error false
  <parse>
    @type "json"
    null_value_pattern "unchanged"
  </parse>
</filter>

Which after the output to Elasticsearch gives:

field	value
...
log_output.WorkflowTemplate/registry-staging/publish-prep.keyword	unchanged
log_output.WorkflowTemplate/registry-staging/registry-image-build	unchanged
log_output.WorkflowTemplate/registry-staging/registry-image-build.keyword	unchanged
log_output.WorkflowTemplate/registry-staging/registry-restart	unchanged
log_output.WorkflowTemplate/registry-staging/registry-restart.keyword	unchanged
...
raw	{"level":"info","ts":"2023-06-29T14:48:48.229Z","msg":"server-side apply completed","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"eks-cluster-konf","namespace":"flux-system"},"namespace":"flux-system","name":"eks-cluster-konf","reconcileID":"06ad0250-3519-4e0a-8889-ffe3d48dd2cf","output":{"ConfigMap/cbd-production/cbd-api-k6bk5m8d82":"unchanged","ConfigMap/cbd-production/cbd-api-logback-config-h97k79tk9d":"unchanged","ConfigMap/cbd-production/cbd-app-hd2b762mk2":"unchanged","ConfigMap/cbd-production/cbd-podium-config-mt9m5dhkk7":"unchanged","ConfigMap/cbd-production/cbd-podium-fuseki-config":"unchanged","ConfigMap/cbd-production/cbd-podium-logback-config-g5fk2fb2kk":"unchanged","ConfigMap/cbd-production/cbd-podium-redis-config":"unchanged",...

Note. The raw field is indented to record the key field log prior to parsing. Here, it is complete and agrees with the input (although truncated here for brevity). Field log is then successfully parsed as json, including all of the log_output.* fields, alas their contents unchanged, again truncated for brevity. Thus I conclude

• fluentd believes my config valid.
• the log is valid and successfully parsed as json
• null_value_pattern is being ignored as fields created with contents unchanged.

[andrew-pickin-epi]

I see this is waiting on me. What more is required?

[daipom]

Sorry! Thanks for your report! I will try to reproduce it!

[daipom]

@andrew-pickin-epi Thanks for sharing the detailed info for reproducing. I can reproduce this.

<source>
  @type tail
  tag test
  path /path/to/sample.json
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<filter test.**>
  @id parser_argo
  @type parser
  key_name message
  reserve_time true
  reserve_data true
  remove_key_name_field true
  inject_key_prefix log_
  emit_invalid_record_to_error false
  <parse>
    @type json
    null_value_pattern unchanged
  </parse>
</filter>

<match test.**>
  @type stdout
</match>

/path/to/sample.json (input)

{"level": "info","ts": "2023-06-29T14:48:48.229Z","msg": "server-side apply completed","controller": "kustomization","controllerGroup": "kustomize.toolkit.fluxcd.io","controllerKind": "Kustomization","Kustomization": {"name": "eks-cluster-konf","namespace": "flux-system"},"namespace": "flux-system","name": "eks-cluster-konf","reconcileID": "06ad0250-3519-4e0a-8889-ffe3d48dd2cf","output": {"ConfigMap/cbd-production/cbd-api-k6bk5m8d82": "unchanged","Deployment/cbd-production/cbd-api": "unchanged","ImagePolicy/cbd-production/cbd-api": "unchanged","Service/cbd-production/cbd-api": "unchanged","WorkflowTemplate/registry-dev/publish-prep": "unchanged"},"revision": "master/a41efdbc4a08041e2ffc0bd092138163477e8a95"}

Input JSON formatted

{
    "level": "info",
    "ts": "2023-06-29T14:48:48.229Z",
    "msg": "server-side apply completed",
    "controller": "kustomization",
    "controllerGroup": "kustomize.toolkit.fluxcd.io",
    "controllerKind": "Kustomization",
    "Kustomization": {
        "name": "eks-cluster-konf",
        "namespace": "flux-system"
    },
    "namespace": "flux-system",
    "name": "eks-cluster-konf",
    "reconcileID": "06ad0250-3519-4e0a-8889-ffe3d48dd2cf",
    "output": {
        "ConfigMap/cbd-production/cbd-api-k6bk5m8d82": "unchanged",
        "Deployment/cbd-production/cbd-api": "unchanged",
        "ImagePolicy/cbd-production/cbd-api": "unchanged",
        "Service/cbd-production/cbd-api": "unchanged",
        "WorkflowTemplate/registry-dev/publish-prep": "unchanged"
    },
    "revision": "master/a41efdbc4a08041e2ffc0bd092138163477e8a95"
}

Result:

2023-07-10 09:13:38 +0900 [info]: #0 fluentd worker is now running worker=0
2023-07-10 09:13:54.074131600 +0900 test: {"log_level":"info","log_ts":"2023-06-29T14:48:48.229Z","log_msg":"server-side apply completed","log_controller":"kustomization","log_controllerGroup":"kustomize.toolkit.fluxcd.io","log_controllerKind":"Kustomization","log_Kustomization":{"name":"eks-cluster-konf","namespace":"flux-system"},"log_namespace":"flux-system","log_name":"eks-cluster-konf","log_reconcileID":"06ad0250-3519-4e0a-8889-ffe3d48dd2cf","log_output":{"ConfigMap/cbd-production/cbd-api-k6bk5m8d82":"unchanged","Deployment/cbd-production/cbd-api":"unchanged","ImagePolicy/cbd-production/cbd-api":"unchanged","Service/cbd-production/cbd-api":"unchanged","WorkflowTemplate/registry-dev/publish-prep":"unchanged"},"log_revision":"master/a41efdbc4a08041e2ffc0bd092138163477e8a95"}

Result(formatted):

{
    "log_level": "info",
    "log_ts": "2023-06-29T14:48:48.229Z",
    "log_msg": "server-side apply completed",
    "log_controller": "kustomization",
    "log_controllerGroup": "kustomize.toolkit.fluxcd.io",
    "log_controllerKind": "Kustomization",
    "log_Kustomization": {
        "name": "eks-cluster-konf",
        "namespace": "flux-system"
    },
    "log_namespace": "flux-system",
    "log_name": "eks-cluster-konf",
    "log_reconcileID": "06ad0250-3519-4e0a-8889-ffe3d48dd2cf",
    "log_output": {
        "ConfigMap/cbd-production/cbd-api-k6bk5m8d82": "unchanged",
        "Deployment/cbd-production/cbd-api": "unchanged",
        "ImagePolicy/cbd-production/cbd-api": "unchanged",
        "Service/cbd-production/cbd-api": "unchanged",
        "WorkflowTemplate/registry-dev/publish-prep": "unchanged"
    },
    "log_revision": "master/a41efdbc4a08041e2ffc0bd092138163477e8a95"
}

[daipom]

This is caused by the nested JSON. null_value_pattern is not applied to the nested JSON values.

This is simply because the JSON parser does not have enough features. (It looks more like a lack of features than a bug.)

[daipom]

You want to discard certain values for nested JSON, right? The current JSON parser can not do that, so we need to think of another way. I can't think of a good way to do it with the normal plugins right away... It would be possible to create such a plugin.

I want to focus this issue on the lack of features of the JSON parser, for the maintenance of Fluentd. For more discussions about how to handle nested JSON, could you please ask about it in the Q&A?

• https://github.com/fluent/fluentd/discussions/categories/q-a