remove_field removes wrong field (dots and underscores are set equal)

[HenryTheSir]

Expected Behavior

pipeline function _remove_field removes field(s) as in 5.0

Current Behavior

remove_field does not difference between _ and '.' in Field name

Possible Solution

Something in 5.1 broke this feature

Steps to Reproduce (for bugs)

1.) Create Pipeline rule

  set_field("test.duplicate", "test-1");
  set_field("test_duplicate", concat(to_string($message.`test.duplicate`),"-2"));
  remove_field("test.duplicate");

2.) observe that test_duplicate is not present

Context

Rules which worked fine in 5.0 got broken with update on 5.1 without any processing error or other indicator

Your Environment

• Graylog Version: 5.1
• Java Version: 17
• OpenSearch Version: 2.7
• MongoDB Version: 5
• Operating System:
• Browser version:

[HenryTheSir]

Changed title and description as error seams to be different than initial thought

[HenryTheSir]

Origin seams to be: https://github.com/Graylog2/graylog2-server/pull/15131

Can this get in the update notes, that the remove_fields function is behaving different:

https://go2docs.graylog.org/5-1/upgrading_graylog/upgrading_to_graylog_5.1.x.htm

Ideally this should be a toggle to use regex or no regex to be backward compatibel with old(er) usages.

[drewmiranda-gl]

I'll defer to the devs on this but my understanding is that because . characters are not allowed in graylog field names, graylog silently replaces dot names in fields with _ so the 2 fields you listed would be treated equally. https://github.com/Graylog2/graylog2-server/issues/13043

[bernd]

@HenryTheSir Thanks for the report. This is a bug.

@drewmiranda-gl Dots are only replaced before ingesting them into OpenSearch. In pipeline functions the fields still have the dot.

[patrickmann]

The problem is that an input string intended to be a field name is being treated as a regex. So a workaround is to specify the appropriate regex, i.e. escape the dot if needed.