Changes to pipeline not applied but simulator works

[maxstoyanov]

Expected Behavior

Changing the referenced grok expression should alter the result of my pipeline.

Even replacing set_field("original_message", message_field); with set_field("event.original", message_field); does not change the result.

Current Behavior

Old version of pipeline and grok expression are used even after reboot.

But adding the debug statement works as expected.

Steps to Reproduce (for bugs)

I know reproducing a bug is essential but I currently lack the time to build a separate instance from scratch and start testing there. So this only "works" on my instance:

1. Setup environment like seen below
2. Create pipeline
3. Change pipeline (referenced grok)

Context

I have logs from my firewall (Cisco ASA) coming to Graylog with an UDP Raw input. Messages based on source ip are rerouted to a dedicated stream (and index set). The stream has a pipeline for message processing attached. I already used a version of this pipeline and started to adjust fields to adhere to the Elastic Common Schema.

Message Processor Configuration is:

1. AWS Instance Name Lookup (disabled)
2. GeoIP Resolver (disabled)
3. Message Filter Chain
4. Pipeline Processor

Your Environment

• Graylog Version: Graylog 3.1.0+aa5175e
• Elasticsearch Version: 6.8.2
• MongoDB Version: 1:3.6.3-0ubuntu1.1
• Operating System: Ubuntu 18.04 Linux 4.15.0-58-generic
• Browser version: Firefox 68.0.2

[jalogisch]

it might be that the . in the field name is the problem - as we do not allow dots in field names.

BUT we need to verify that.

[maxstoyanov]

I tried that but no change. I'll try to isolate the problem as soon as possible. (But will take a while due to different project priorities.)

[patrickmann]

@maxstoyanov reports that the issue is no longer reproducible in more recent versions of GL. Additionally, we now automatically replace a "." character in the field name with "_".