splunk-connect-for-syslog

splunk-connect-for-syslog copied to clipboard

All events from vcenter are showing up with sourcetype=vmware:esxlog: and sc4s_class=esx

It also appears sc4s is prefixing events with

- - - - - [meta sequenceId="1996712"]

I have a pcap I can share via Slack DM just let me know who to send it to.

@zyphermonkey please share PCAP file with me in DM on Slack (its Mateusz Pierzchała)

Also please tell me what sourcetype and you are expecting to be please.

The sourcetype should be vmware:vclog:${PROGRAM} per the docs.

pcap incoming

Hi, I created postfilter for you as your messages are not compliant with RFCs. You can place it /opt/sc4s/local/config/app_parsers/filters.

`# app-dest-rewrite-vmware_vclog.conf block parser app-dest-rewrite-vclog_postfilter() { channel {

    filter {
        match("vsphere" value("fields.sc4s_product") type(string))
        and match("vmware" value("fields.sc4s_vendor") type(string))
        and match("" value(PROGRAM));        };
    if {
        parser {
            regexp-parser(
                prefix(".tmp.")
                patterns('(?<timestamp>\d{4}-\d{2}-\d{2}[T\s][\d\:\.]{8,15}([\+\-\s,][\d\:]{3,5}|Z|))\s(?<loglevel>\w+)\s(?<program>[a-zA-Z0-9]+)\[[0-9]{5}\](?<message>.*)')
                template("$MESSAGE")
            );
        };

        parser {
            date-parser-nofilter(
                format('%Y-%m-%dT%H:%M:%S.%f%z')
                template("${.tmp.timestamp}")
            );
        };
       rewrite {
           set("${.tmp.message}" value("MESSAGE"));
        };
       rewrite {
           set("${.tmp.program}", value(".PROGRAM"));
        };
       rewrite {
       r_set_splunk_dest_default(
            source('vmware:vclog:${.tmp.program}')
            sourcetype('vmware:vclog:${.tmp.program}')
            class('vc')

);
      };
    };
};

};`

@rjha-splunk

I like to get to the root of this.

Okay so according the to the Splunk Docs they want you to set the log template to

$template vclogtemplate,"%syslogtag% %rawmsg%"
Which is when you get logs that are most likely not RFC compliant.

If I use the vCenter gui to enable syslog it sets it to

RSYSLOG_SyslogProtocol23Format

Which appears to be more inline with what SC4S is expecting.
It doesn't matter which format we ultimately use we just need to make sure SC4S and the VMware add-on are in sync.

Examples

template(name="RSYSLOG_SyslogProtocol23Format" type="string" string="<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n")

**tcpdump** <134>1 2022-10-28T11:59:51.142896+00:00 icam-vc-p01 vpxd-main - - - 2022-10-28T11:59:51.141Z info vpxd[06444] [Originator@6876 sub=vpxLro opID=sps-Main-188474-39-958391-d6] [VpxLRO] -- FINISH session[5293a3cd-4a7c-bcdf-910e-65c7b5d1792b]52708015-2b81-a800-3214-82b53b340867

**Splunk** - - vpxd-main - - [meta sequenceId="159157423"]2022-10-28T11:59:51.141Z info vpxd[06444] [Originator@6876 sub=vpxLro opID=sps-Main-188474-39-958391-d6] [VpxLRO] -- FINISH session[5293a3cd-4a7c-bcdf-910e-65c7b5d1792b]52708015-2b81-a800-3214-82b53b340867

$template vclogtemplate,"%syslogtag% %rawmsg%"

**tcpdump** vpxd 2022-10-28T12:17:24.306Z warning vpxd[06499] [Originator@6876 sub=vmomi.soapStub[4666] opID=TaskLoop-host-13224] SOAP request returned HTTP failure; <SSL(<io_obj p:0x00007fdf288bd558, h:81, <TCP '10.108.3.23 : 35976'>, <TCP '10.108.3.46 : 443'>>), /vpxa>, method: waitForUpdates; code: 500(Internal Server Error)

**Splunk** - - - - - [meta sequenceId="161073524"]vpxd 2022-10-28T12:17:24.306Z warning vpxd[06499] [Originator@6876 sub=vmomi.soapStub[4666] opID=TaskLoop-host-13224] SOAP request returned HTTP failure; <SSL(<io_obj p:0x00007fdf288bd558, h:81, <TCP '10.108.3.23 : 35976'>, <TCP '10.108.3.46 : 443'>>), /vpxa>, method: waitForUpdates; code: 500(Internal Server Error)