Kong's log system does not allow you to set fields with dots

[john-wd]

Is there an existing issue for this?

• [X] I have searched the existing issues

Kong version ($ kong version)

3.2.2.2

Current Behavior

When I set a custom field to the file-log plugin containing dots, it automatically unpacks the name into nested fields. For example, when I try to set Google's magic trace field logging.googleapis.com/trace, it unpacks as:

{
  logging:
    googleapis:
      com/trace: "my-trace"
}

which does not integrate with google cloud at all.

Expected Behavior

When I set field with dots, I expect it to keep dots

{
  logging.googleapis.com/trace: "my-trace"
}

Steps To Reproduce

1. Configure file-log setting a custom field containing dots

plugins:
- name: file-log
  config:
    path: "/tmp/file.log"
    custom_fields_by_lua:
      "logging.googleapis.com/trace": "return kong.request.get_header('x-cloud-trace-context')"

2. Hit any exposed endpoint
3. See in /tmp/file.log that this field was broken into a nested document

Anything else?

This seems to be the default behavior judging by these lines, but I'd expect to be able to integrate with my cloud provider.

Additional info:

• Deployed with public helm chart
• Tag: 3.2.2.2-ubuntu

[StarlightIbuki]

Thanks for reporting this. We can locate the logic handling the dots here. Maybe we could introduce an escape for the dot. We will discuss this.

[hanshuebner]

Create internal ticket KAG-2601

[github-actions[bot]]

This issue is marked as stale because it has been open for 14 days with no activity.

[StarlightIbuki]

This should be easy to fix: for example, we could use % to escape . ("%." for . and "%%" for %). But this would be a breaking change. This could fit more in the next major version. I'm trying to confirm this. Before we have such improvement, maybe we could workaround with a custom plugin.

[mheap]

Why use % rather than something more standard such as \ ?

[StarlightIbuki]

Why use % rather than something more standard such as \ ?

% is just used to illustrate. To make a choice I will pick a character that is less likely to occur in the name.

[ms2008]

The issue is sort of conditionally fixed. That is, when you send a JSON request it works (after this PR #11091), but when you send a form data request it doesn't work because the variable name in the form data contains the . symbol, which is recognized by Kong as nested keys, and then normalize dotted keys in objects which will cause schema violation.

[john-wd]

@ms2008 will this fix be a either/or situation in which either all dots in, say custom_fields_by_lia, will be escaped or all of them will break into a nested table?

[ms2008]

@catbro666 What about your thoughts?

[catbro666]

@ms2008 This should be a separate issue from https://github.com/Kong/kong/pull/11091 . That one is about validation failure when the key contains dots, while this one is because the dot character has a special meaning in the keys of custom_fields_by_lua but there are no ways to escape it.