mail.log dates in the future

[crazy-man]

Hello,

Today I discovered some entries in ES dated 2017/12/31 (in the future). Since mail.log does not contain year, moment.js adds current year. Those entries were parsed in 2017. I wrote some simple workaround for that: (postfix-doc.js line 41:)

  var sys_date = moment(sys.date, 'MMM DD HH:mm:ss');
  if (sys_date.isAfter(moment(), 'day')) {
    sys_date = sys_date.subtract(1, 'years');
  }
  parsed.date = sys_date.format();

It just checks if parsed date is in the future( I used 'day' granularity to avoid problems with DST +-1 hour)

[msimerson]

I believe you mean, those entries were parsed in 2016?

When dealing with edge cases, an important considerations is not creating new edge cases, as this change has the potential to do. There might be other edge cases where moment.js parses a syslog date incorrectly, and subtracting a year may not be the right solution there. I'm thinking the optimal solution is:

a) write a test that covers the known case(s) when this happens. Near as I can tell, the only day this will ever happen is on 12/31. b) constrain the code check to just the known edge cases (only run on Dec 31).

[crazy-man]

Entries happened (were created) in 2016, but log-ship-elastic-postfix parsed (processed) them in 2017.

I agree that guessing a year is some kind of heuristics here. We just can't know correct year. But it is reasonable to assume that if a parsed year is in the future, it can't be correct anyway.

If moment.js can't parse date correctly for some reason, we get wrong date anyway, so it doesn't really matter in that case.

a) In a regular situation, this seems to be only happening on a small amount of events on 12/31, but there can be edge cases, when for example log-ship-elastic-postfix was down for a while and after some time you run it to import logs from the past year. b) According to scenario described above(importing logs in bulk) any date in the future(not just 12/31) is 100% incorrect. Subtracting 1 year doesn't make it correct either (those could be 2 year old logs).