watched file might be removed while trying to read

[kares]

the plugin in certain rare cases runs into a concurrency problem, while trying to read :

[ERROR][filewatch.readmode.handlers.readfile] controlled_read: general error reading file {
"path"=>"/logstash/data/path/to.json", 
"error"=>"#<NoMethodError: undefined method `increment_position' for nil:NilClass>", 
"backtrace"=>[
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.11/lib/filewatch/sincedb_collection.rb:144:in `increment'", 
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.11/lib/filewatch/read_mode/handlers/read_file.rb:41:in `block in controlled_read'", 
"org/jruby/RubyArray.java:1792:in `each'", 
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.11/lib/filewatch/read_mode/handlers/read_file.rb:37:in `block in controlled_read'", 
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.11/lib/filewatch/read_mode/handlers/read_file.rb:32:in `controlled_read'", 
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.11/lib/filewatch/read_mode/handlers/read_file.rb:11:in `block in handle_specifically'", 
"org/jruby/RubyKernel.java:1425:in `loop'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.11/lib/filewatch/read_mode/handlers/read_file.rb:8:in `handle_specifically'"
]}

... the watched file entry got removed in this case but the plugin still triggered a read attempt. we need to investigate what all potential consequences are around such issues (e.g. if they could cause event loss from other parts).

[insukcho]

I think I also hit a similar situation.

[ERROR][filewatch.tailmode.handlers.grow] 
read_to_eof: general error reading D:/xxxxx.csv 
{
	"error"=>"#<NoMethodError: undefined method `increment_position' for nil:NilClass>", 
	"backtrace"=>[
		"C:/ELK/logstash-7.1.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/sincedb_collection.rb:144:in `increment'", 
		"C:/ELK/logstash-7.1.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/tail_mode/handlers/base.rb:58:in `block in controlled_read'", 
		"org/jruby/RubyArray.java:1792:in `each'", 
		"C:/ELK/logstash-7.1.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/tail_mode/handlers/base.rb:55:in `block in controlled_read'"]
}

What is the reason for these logs? Any idea @kares ?

[kares]

Thanks Chris, definitely a (concurrency) bug but I believe it was handled in plugin's most recent releases. You might want to bin\logstash-plugin update logstash-input-file to install the latest (4.2.1) version. Please share if you have any parameter customizations, esp. the numeric parameters, for your plugin instance.

[sathishkumarbalasubramani]

Hi @kares Could you please help me to download the version 4.2.1? I am new to ELK. Thanks.