[Filebeat - v7.17.2] Using the close.reader.on_eof: true option always results Error - Read line error: EOF

[sinhagithub]

Using the close.reader.on_eof: true option always results in the Error at the EOF: So, am I missing some configuration here, or this a bug (as EOF should not be reported as error):

log.Errorf("Read line error: %v", err)

2022-04-14T18:59:16.106Z ERROR [input.filestream] filestream/input.go:322 Read line error: EOF {"id": "cdr-dqi", "source": "filestream::cdr-dqi::native::3561807-64769", "path": "/opt/sdf/cdr/ts_20220414_185910.cdr", "state-id": "native::3561807-64769"}

filebeat.yaml

# CDR
- type: filestream

  # Change to true to enable this input configuration.
  enabled: true
  id: cdr-dqi

  # Tuning options
  close.reader.on_eof: true
  close_inactive: 5m
  ignore_older: 6m
  clean_inactive: 7m

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /opt/sdf/cdr/ts_[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_*.cdr

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  fields:
    logtype: cdr

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  exclude_lines: ["^CDR Header", "^CDR Footer"]

File content:

[sdfrun@dqi-ssi2-dqi-86654c8476-52ctb cdr]$ tail /opt/sdf/cdr/ts_20220414_185910.cdr ... DQI,F,1404202218:59:10.639,,unknownIMSI,,43,SOAP:SOAP request type error: start tag not found,0,N/A,N/A,2321,1,10.32.22.15 DQI,F,1404202218:59:10.639,,unknownIMSI,,43,SOAP:SOAP request type error: start tag not found,0,N/A,N/A,2321,50,200.100.100.100 CDR Footer [sdfrun@dqi-ssi2-dqi-86654c8476-52ctb cdr]$

[box4box]

I have the same problem with usage of close.reader.on_eof: true with filebeat 7.17.3

I found the issue. With filebeat 7.16.x (and higher) the close.reader.on_eof had been renamed. Before, with filebeat 7.15.x (and lower) the nameing had been close.reader.eof. Problem is, that the usage of close.reader.on_eof results into the above mentioned error and therefore it is not working.

Workaround: (at least working for me) With filebeat 7.16.x (and higher) please still use close.reader.eof, and not close.reader.on_eof This workaround may work, until it got fixed.

Here the documentation links: filebeat 7.15.x for close.reader.eof: LINK filebeat current for close.reader.on_eof: LINK

@filebeat team:

• The current documentation need to be adjusted, that it is eof and not on_eof
• Or the on_eof need to be correctly implemented into the filebeat.

[kvch]

@sinhagithub It has been fixed in https://github.com/elastic/beats/pull/31109

@box4box The documentation for 7.15 has not been updated. The correct option name has always been on_eof. You do not see the error message in the logs because the setting is not enabled. You should use on_eof in all versions of filestream.

[kesslerm]

The issue appears to be fixed only in the current release series, but not in the 7.17 series against which it was originally reported. Can this be backported to 7.17, please?

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)