New patterns

[leweafan]

I added several patterns so may be it'll be helpful.

POSTFIX_CLEANUP_REPLACE %{POSTFIX_QUEUEID:postfix.queueid}: replace: header Message-(Id|ID): <%{NOTSPACE}> from %{POSTFIX_CLIENT_INFO}; %{POSTFIX_KEYVALUE_DATA:postfix.keyvalue_data}: Message-(Id|ID): <%{NOTSPACE}>

POSTFIX_SMTP_SSLAUTHERR %{POSTFIX_QUEUEID:postfix.queueid}: SASL authentication failed; server %{POSTFIX_RELAY_INFO} said: %{GREEDYDATA:postfix.smtp_response}

POSTFIX_CLEANUP %{POSTFIX_CLEANUP_MILTER}|%{POSTFIX_CLEANUP_REPLACE}|%{POSTFIX_WARNING}|%{POSTFIX_KEYVALUE} POSTFIX_SMTP %{POSTFIX_SMTP_DELIVERY}|%{POSTFIX_SMTP_CONNERR}|%{POSTFIX_SMTP_SSLAUTHERR}|%{POSTFIX_SMTP_SSLCONNERR}|%{POSTFIX_SMTP_LOSTCONN}|%{POSTFIX_SMTP_TIMEOUT}|%{POSTFIX_SMTP_RELAYERR}|%{POSTFIX_TLSCONN}|%{POSTFIX_WARNING}|%{POSTFIX_SMTP_UTF8}|%{POSTFIX_TLSVERIFICATION}

POSTFIX_POSTMAP %{POSTFIX_WARNING} POSTFIX_SCRIPT %{POSTFIX_WARNING}

Grok config file part:

if [program] =~ /^postfix.*\/postmap$/ {
    grok {
        patterns_dir   => "/etc/logstash/patterns"
        match          => [ "postfix.full_message", "^%{POSTFIX_POSTMAP}$" ]
        tag_on_failure => [ "_grok_postfix_postmap_nomatch", "_grokparsefailure" ]
        add_tag        => [ "_grok_postfix_success" ]
    }
} else if [program] =~ /^postfix.*\/postfix-script$/ {
    grok {
        patterns_dir   => "/etc/logstash/patterns"
        match          => [ "postfix.full_message", "^%{POSTFIX_SCRIPT}$" ]
        tag_on_failure => [ "_grok_postfix_script_nomatch", "_grokparsefailure" ]
        add_tag        => [ "_grok_postfix_success" ]
    }
}

Also there are several postfix.smtp_response patterns:

if [program] =~ /^postfix.*\/smtp$/ {
  grok {
      patterns_dir   => "/etc/logstash/patterns"
      match          => [ "postfix.full_message", "^%{POSTFIX_SMTP}$" ]
      tag_on_failure => [ "_grok_postfix_smtp_nomatch", "_grokparsefailure" ]
      add_tag        => [ "_grok_postfix_success" ]
  }
  if "postfix.smtp_response" {
    grok {
        patterns_dir   => "/etc/logstash/patterns"
        match => {
          "postfix.smtp_response" => [
            "^host %{NOTSPACE} said: %{POSTFIX_STATUS_CODE:postfix.status_code}",
            "%{POSTFIX_STATUS_CODE:postfix.status_code}(-| )%{POSTFIX_STATUS_CODE_ENHANCED:postfix.status_code_enhanced} %{POSTFIX_WARNING_LEVEL:postfix.message_level}: %{GREEDYDATA:postfix.message}",
            "%{POSTFIX_STATUS_CODE:postfix.status_code}(-| )%{POSTFIX_STATUS_CODE_ENHANCED:postfix.status_code_enhanced} %{GREEDYDATA:postfix.message}",
            "%{POSTFIX_STATUS_CODE:postfix.status_code} %{GREEDYDATA:postfix.message}"
          ]
        }
        tag_on_failure => [ "_grok_postfix_smtp_response_nomatch", "_grokparsefailure" ]
        add_tag        => [ "_grok_postfix_success" ]
    }
  }
}

[Devastate-D]

Hello, can you show which log parses this patterns? Thanks

[leweafan]

POSTFIX_CLEANUP_REPLACE example:

Aug  6 16:46:42 mr01 postfix/cleanup[7548]: 127DC5E156: replace: header Message-Id: <[email protected]> from unknown[10.10.10.10]; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<host1.com>: Message-ID: <[email protected]>

POSTFIX_SMTP_SSLAUTHERR example:

Aug  6 16:47:42 mr01 postfix/mr/smtp[11361]: D0F29603B4: SASL authentication failed; server [xxx.yyy.com](http://xxx.yyy.com/)[10.10.10.10] said: 535 5.7.8 Error: authentication failed: authentication failure

POSTFIX_POSTMAP example:

Aug  6 16:17:02 mr01 postfix/postmap[12924]: warning: /etc/postfix/conf.d/users.db: duplicate entry: “[email protected]"

POSTFIX_SCRIPT examples:

Aug  4 14:23:01 mr01 postfix/postfix-script[895]: warning: symlink leaves directory: /etc/postfix/./makedefs.out
Aug  4 14:23:02 mr01 postfix/postfix-script[8442]: warning: /var/spool/postfix/lib/x86_64-linux-gnu/libnss_nisplus-2.28.so and /lib/x86_64-linux-gnu/libnss_nisplus-2.28.so differ

postfix.smtp_response 1st pattern:

Aug  6 16:59:34 mr01 postfix/smtp[15280]: 0D75B5E169: to=<[[email protected]](mailto:[email protected])>, relay=ASPMX.L.GOOGLE.COM[66.102.1.27]:25, delay=2.5, delays=0.01/0/2.3/0.11, dsn=5.1.1, status=bounced (host ASPMX.L.GOOGLE.COM[66.102.1.27] said: 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the recipient's email address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1  https://support.google.com/mail/?p=NoSuchUser n14-20020a5d598e000000b00220748183cesi5229536wri.43 - gsmtp (in reply to RCPT TO command))
Aug  6 16:59:32 mr01 postfix/smtp[16164]: 0730C5E156: to=<[[email protected]](mailto:[email protected])>, relay=[zzz.mail.com](http://zzz.mail.com/)[10.10.10.10]:25, delay=1, delays=0.1/0/0.61/0.28, dsn=5.0.0, status=bounced (host [zzz.mail.com](http://zzz.mail.com/)[10.10.10.10] said: 550 Message was not accepted -- invalid mailbox.  Local mailbox [email protected] is unavailable: account is disabled (in reply to end of DATA command))
Aug  6 11:23:03 mr01 postfix/smtp[20833]: 340A112011D: to=<[[email protected]](mailto:[email protected])>, relay=[zzz.mail.com](http://zzz.mail.com/)[10.10.10.10]:25, delay=97103, delays=97103/0.04/0.51/0, dsn=4.7.1, status=deferred (host [zzz.mail.com](http://zzz.mail.com/)[10.10.10.10] refused to talk to me: 554 5.7.1 You are not allowed to connect.)

postfix.smtp_response 2nd pattern:

Will send later when find it.

postfix.smtp_response 3rd pattern:

Aug  6 16:58:03 mr01 postfix/smtp[15453]: D96121200FF: host aspmx.l.google.com[64.233.164.27] said: 450-4.2.1 The user you are trying to contact is receiving mail too quickly. 450-4.2.1 Please resend your message at a later time. If the user is able to 450-4.2.1 receive mail at that time, your message will be delivered. For more 450-4.2.1 information, please visit 450 4.2.1  https://support.google.com/mail/?p=OverReceiveLimit z19-20020a2e3513000000b0025e46bd4d56si3707193ljz.374 - gsmtp (in reply to RCPT TO command)

postfix.smtp_response 4th pattern:

Aug  6 17:11:51 mr01 postfix/mr/smtp[15439]: 106A9603C4: to=<[email protected]>, relay=zzz.mail.com[10.10.10.10]:25, delay=1.2, delays=0.05/0/0.93/0.21, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 1C9B35E168)

[Devastate-D]

thank you for the answer, I have not met such rare errors, thank you again

[whyscream]

Some comments:

• POSTFIX_CLEANUP_REPLACE parses one very specific type of output that is produced by cleanup: a REPLACE action on a header line that contains a Message-ID. All other cleanup actions are ignored. As the output of cleanup can be very diverse, or even the output of a REPLACE action by clean, I'd opt for not parsing it. If you actually need to extract the original and replaced Message-ID from this logline, that should be part of your local setup.
• POSTFIX_SMTP_SSLAUTHERR: This looks fine, I'll create a PR for this.
• POSTFIX_POSTMAP: Fine.
• POSTFIX_SCRIPT: Fine.
• In the past there have been questions regarding parsing the response of a remote SMTP server, as is done with postfix.smtp_response. I opted to not do that, for several reasons. 1) There's no way to know whether a remote server is actually Postfix, so parsing the server response will result in various stages of success. This might be confusing. 2) In the end, this repo is about parsing the logs of the local server. If we get similar-looking data from remote server data in parsed fields, this also might confuse people.

[whyscream]

FInally came around to add these patterns, sorry about the delay.