Concat plugin makes some log messages disappear

[artm]

Problem

A simple rule: if a line starts with two spaces or is empty it is continuation of a previous rule.

I set continuous_line_regexp to /^( |$)/.

Now most logs disappear, but somehow 3 events every hour don't.

Steps to replicate

log that looks like

 [OK] All scheduled tasks are executed!                                         

 [OK] Job "'/usr/local/bin/php' '-d display_errors=stderr'                      
      '/var/www/bin/console' '--env=prod' 'cron:run'" executed successfully.    
                                                                                
      time: 0.099830150604248                                                   
                                                                                


 [OK] All scheduled tasks are executed!                                         

 [OK] Job "'/usr/local/bin/php' '-d display_errors=stderr'                      
      '/var/www/bin/console' '--env=prod' 'cron:run'" executed successfully.    
                                                                                
      time: 0.0097789764404297                                                  
                                                                                
                                                                                
 [OK] Job "'/usr/local/bin/php' '-d display_errors=stderr'                      
      '/var/www/bin/console' '--env=prod' 'XXXX:YYYY'"       
      executed successfully.                                                    

only the last event is present (consistently every time it happerns) in the result correctly concatenated all the rest are missing.

Found another event that survives:

 [OK] Job "XXXX\YYYY_redacted_long_string" executed             
      successfully.    

I can't figure out the pattern. Maybe the fact that one broken event contains just a singe start line followed by what I hoped would be a continuation line and the other broken event contains an empty line in the middle. But even if pattern doesn't match I would expect the log lines come out of the plugin incorrectly concatenated in stead of disappearing.

Your environment

this plugin is running as part of kubernetes logging operator.

[artm]

I was mistaken.

• the message [OK] All scheduled tasks are executed! is surviving the plugin, it was filtered out in the dashboard
• the empty lines don't get matched and concatenated because these events simply don't contain the field I'm matching and concatenating

so the only message that gets lost seems to be:

 [OK] Job "'/usr/local/bin/php' '-d display_errors=stderr'                      
      '/var/www/bin/console' '--env=prod' 'cron:run'" executed successfully.    
                                                                                
      time: 0.0097789764404297                                                  
                                                                                
                                                                               

[artm]

I now see that users keep reporting similar problems, like #109, #94 and various other related to time outs and disappearing log records. Also a PR #121 seems related as our events that get swallowed have the same timestamp as the 1 that remains. I conclude that this plugin isn't ready to be used in production.