unexpected error error="undefined method `[]=' for nil:NilClass"

[CheyiLin]

We got the following error when sending messages from filebeat to fluentd:

unexpected error error="undefined method `[]=' for nil:NilClass"

With the traceback:

/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/fluent/plugin/in_beats.rb:103:in `block (3 levels) in run'
/var/lib/gems/2.3.0/gems/fluentd-0.14.20/lib/fluent/plugin/parser_json.rb:65:in `rescue in parse'
/var/lib/gems/2.3.0/gems/fluentd-0.14.20/lib/fluent/plugin/parser_json.rb:61:in `parse'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/fluent/plugin/in_beats.rb:102:in `block (2 levels) in run'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:386:in `data'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:361:in `block (2 levels) in read_socket'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:373:in `ack_if_needed'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:357:in `block in read_socket'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:246:in `json_data_payload'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:163:in `feed'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:296:in `compressed_payload'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:163:in `feed'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:342:in `read_socket'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:319:in `run'
/var/lib/gems/2.3.0/gems/fluent-plugin-beats-0.1.2/lib/fluent/plugin/in_beats.rb:97:in `block in run'
/var/lib/gems/2.3.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:331:in `run_task'
/var/lib/gems/2.3.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (3 levels) in create_worker'
/var/lib/gems/2.3.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in `loop'
/var/lib/gems/2.3.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in `block (2 levels) in create_worker'
/var/lib/gems/2.3.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in `catch'
/var/lib/gems/2.3.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in `block in create_worker'

The version info:

• filebeat 5.6.0
• fluentd 0.14.20
• fluent-plugin-beats 0.1.2

The filebeat config:

filebeat.prospectors:
- input_type: log
  paths:
    - /tmp/logs/*.log
output.logstash:
  enabled: true
  hosts: ["fluentd:5044"]
logging.level: debug

The fluentd config:

<source>
  @type beats
  port 5044
  tag mytag.beats
  format json
  time_key @timestamp
  time_format %Y-%m-%dT%H:%M:%S.%L%Z
</source>

The filebeat debug log shows that the message failed to send:

filebeat_1      | 2017/09/13 11:15:20.033374 log_file.go:84: DBG  End of file reached: /tmp/logs/test.log; Backoff now.
filebeat_1      | 2017/09/13 11:15:21.977556 spooler.go:89: DBG  Flushing spooler because of timeout. Events flushed: 1
filebeat_1      | 2017/09/13 11:15:21.977825 client.go:214: DBG  Publish: {
filebeat_1      |   "@timestamp": "2017-09-13T11:15:17.032Z",
filebeat_1      |   "beat": {
filebeat_1      |     "hostname": "c80bd1b1ef19",
filebeat_1      |     "name": "c80bd1b1ef19",
filebeat_1      |     "version": "5.6.0"
filebeat_1      |   },
filebeat_1      |   "input_type": "log",
filebeat_1      |   "message": "12345 aaaaa",
filebeat_1      |   "offset": 12,
filebeat_1      |   "source": "/tmp/logs/test.log",
filebeat_1      |   "type": "log"
filebeat_1      | }
filebeat_1      | 2017/09/13 11:15:21.977846 output.go:109: DBG  output worker: publish 1 events
filebeat_1      | 2017/09/13 11:15:21.977869 sync.go:53: DBG  connect
filebeat_1      | 2017/09/13 11:15:21.983998 sync.go:107: DBG  Try to publish 1 events to logstash with window size 10
filebeat_1      | 2017/09/13 11:15:21.985179 client.go:194: DBG  handle error: EOF
filebeat_1      | 2017/09/13 11:15:21.985239 client.go:110: DBG  closing
filebeat_1      | 2017/09/13 11:15:21.985533 sync.go:78: DBG  0 events out of 1 events sent to logstash. Continue sending
filebeat_1      | 2017/09/13 11:15:21.985593 sync.go:58: DBG  close connection
filebeat_1      | 2017/09/13 11:15:21.985631 sync.go:85: ERR Failed to publish events caused by: EOF
filebeat_1      | 2017/09/13 11:15:21.985646 single.go:91: INFO Error publishing events (retrying): EOF
filebeat_1      | 2017/09/13 11:15:21.985652 sync.go:58: DBG  close connection
filebeat_1      | 2017/09/13 11:15:21.985657 single.go:156: DBG  send fail
fluentd_1  | 2017-09-13 11:15:21 +0000 [error]: #0 unexpected error error="undefined method `[]=' for nil:NilClass"
....

Is there anything that I missed or did wrong? Thanks.

[triha74]

We have the exact same issue

[repeatedly]

@triha74 Do you use same config with this issue?

The problme of this issue is using invalid config for message. They use "format json" in in_beats but their beats send non json message, "message": "12345 aaaaa" from log. This is why above error happens. Do you do same mistake or use other configuration?

[rljie]

How do you get filebeat's tags

[tanintem]

Any updates? I got the same error.