graylog2-server GELF message has empty mandatory "short

trafficstars

Current Behavior

I've a lot of errors in log:

GELF message has empty mandatory "short_message" field.

Possible Solution

It is known problem of docker https://github.com/docker/for-linux/issues/354

Your Environment

Graylog Version: 3
Elasticsearch Version: 6
MongoDB Version: 3
Operating System: Ubuntu 18.04
Browser version: -

Mar 22 '19 10:03 mahnunchik

Possible related to https://github.com/Graylog2/graylog2-server/issues/6488

Sep 24 '19 13:09 chuegel

I am getting the same after migration to Graylog 3.1.

Nov 15 '19 14:11 denis-zvonov

I can confirm this Bug. But I have other circumstances. Using nxlog-ce to send GELF_TCP. I also wrote the GELF_TCP-Message to a file an got this:

{"version":"1.1",
"_EventReceivedTime":"2020-08-24 11:04:02",
"_SourceModuleName":"redmine_production",
"_SourceModuleType":"im_file",
"_SourceName":"Redmine Production",
"host":"VEP01",
"short_message":"Started GET \"**********" for 127.0.0.1 at 2020-08-24 11:04:01 +0200\r\nProcessing by TimeLoggersController#render_menu as */*\r\n  Parameters: {\"_\"=>\"1598252400620\"}\r\n  Current user: **** (id=**)\r\n  Rendered ******************** (11.3ms)\r\nCompleted 200 OK in 20ms (Views: 9.9ms | ActiveRecord: 5.7ms)\r",
"timestamp":1598259842,
"level":6}

As you can see the short_message is there. But graylog says:

2020-08-24T11:04:02.121+02:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=**********, journalOffset=260784, codec=gelf, payloadSize=233, timestamp=2020-08-24T09:04:02.120Z, remoteAddress=/192.168.1.***:61723} on input <5f40331381d87961ab292a56>.
2020-08-24T11:04:02.121+02:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=c0fcfc81-e5e8-11ea-aa55-00505681aa85, journalOffset=260784, codec=gelf, payloadSize=233, timestamp=2020-08-24T09:04:02.120Z, remoteAddress=/192.168.1.***:61723}
java.lang.IllegalArgumentException: GELF message <*****************> (received from <192.168.1.***:61723>) has empty mandatory "short_message" field.
	at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:258) ~[graylog.jar:?]
	at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140) ~[graylog.jar:?]
	at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
	at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
	at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]

Also noteable might be, the message is prcessed correctly by graylog.

Maybe this helps to solve this issue.

Aug 24 '20 09:08 mboremski

My graylog logs are filled with these message. It's about 10K lines each minute.

Maybe recreate the input is a workaround.

Sep 07 '20 06:09 h4wkmoon

I have the same problem with my Graylog 4.0.11+e4e88a4, codename Noir. Any update?

Here are the logs :

2022-01-14 09:48:04,082 ERROR [DecodingProcessor] - Unable to decode raw message RawMessage{id=119053b1-751f-11ec-96f4-7af235b1e3ab, journalOffset=198692353, codec=gelf, payloadSize=514, timestamp=2022-01-14T09:48:04.075Z, remoteAddress=/10.40.0.0:19396} on input <607e93166eddf7582244c00f>. - {} 2022-01-14 09:48:04,082 ERROR [DecodingProcessor] - Error processing message RawMessage{id=119053b1-751f-11ec-96f4-7af235b1e3ab, journalOffset=198692353, codec=gelf, payloadSize=514, timestamp=2022-01-14T09:48:04.075Z, remoteAddress=/10.40.0.0:19396} - {} java.lang.IllegalArgumentException: GELF message <119053b1-751f-11ec-96f4-7af235b1e3ab> (received from <10.40.0.0:19396>) has empty mandatory "short_message" field. at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:258) ~[graylog.jar:?] at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140) ~[graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:149) ~[graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:90) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?] at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?] at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]