Add support for numerous additional rsyslog formats

[joachimmetz]

Per https://www.rsyslog.com/doc/v8-stable/configuration/templates.html there are numerous additional rsyslog formats. Extend the syslog parser to support these additional formats

• create a test file
• extend syslog parser

[jonathan-greig]

https://www.rsyslog.com/doc/v8-stable/configuration/templates.html lists thirteen different default templates. Two are already supported by Plaso and six didn't seem to work in my testing VM (Debian 10, rsyslog v8.1901.0):

Format Name	Status
RSYSLOG_TraditionalFileFormat	Already supported
RSYSLOG_FileFormat	Already supported
RSYSLOG_TraditionalForwardFormat	Worked in rsyslog.conf
RSYSLOG_SysklogdFileFormat	Worked in rsyslog.conf
RSYSLOG_ForwardFormat	Worked in rsyslog.conf
RSYSLOG_SyslogProtocol23Format	Worked in rsyslog.conf
RSYSLOG_DebugFormat	Worked in rsyslog.conf
RSYSLOG_WallFmt	Error: Could not find template 0 'RSYSLOG_WallFmt'
RSYSLOG_StdUsrMsgFmt	Error: Could not find template 0 'RSYSLOG_StdUsrMsgFmt'
RSYSLOG_StdDBFmt	Error: Could not find template 0 'RSYSLOG_StdDBFmt'
RSYSLOG_StdPgSQLFmt	Error: Could not find template 0 'RSYSLOG_StdPgSQLFmt'
RSYSLOG_spoofadr	Error: Could not find template 0 'RSYSLOG_spoofadr'
RSYSLOG_StdJSONFmt	Error: Could not find template 0 'RSYSLOG_StdJSONFmt'

The rsyslog source does seem to reference these other templates without the RSYSLOG_ prefix but I couldn't get them to work. Also some of these formats don't have timestamps (RSYSLOG_StdUsrMsgFmt) or seem unlikely to be used in prod (RSYSLOG_DebugFormat).

I think it makes the most sense to focus on default templates that work and are likely to be used in production logging:

• RSYSLOG_TraditionalForwardFormat
• RSYSLOG_SysklogdFileFormat
• RSYSLOG_ForwardFormat
• RSYSLOG_SyslogProtocol23Format

[jonathan-greig]

RSYSLOG_SysklogdFileFormat appears similar enough that the existing parser for RSYSLOG_TraditionalFileFormat can parse, compare:

template(name="RSYSLOG_TraditionalFileFormat" type="string"
     string="%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n")

with:

template(name="RSYSLOG_SysklogdFileFormat" type="string"
     string="%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%\n")

[joachimmetz]

@jonathan-greig I've assigned the issue to you, feel free to close if there is nothing remaining to do

[joachimmetz]

@jonathan-greig was this completed or are there still certain rsyslog formats not supported?

[jonathan-greig]

I didn't add the debug format or any formats which weren't actually supported by rsyslog (despite the documentation saying they were).

Format name	Status
RSYSLOG_DebugFormat	Didn't add support. Docs specifically say "used for troubleshooting property problems. [...] Do not use for production or remote forwarding."
RSYSLOG_WallFmt	Not actually supported by rsyslog: "Error: Could not find template 0 'RSYSLOG_WallFmt'"
RSYSLOG_StdUsrMsgFmt	Not actually supported by rsyslog: "Error: Could not find template 0 'RSYSLOG_StdUsrMsgFmt'"
RSYSLOG_StdDBFmt	Not actually supported by rsyslog: "Error: Could not find template 0 'RSYSLOG_StdDBFmt'"
RSYSLOG_StdPgSQLFmt	Not actually supported by rsyslog: "Error: Could not find template 0 'RSYSLOG_StdPgSQLFmt'"
RSYSLOG_spoofadr	Not actually supported by rsyslog: "Error: Could not find template 0 'RSYSLOG_spoofadr'"
RSYSLOG_StdJSONFmt	Not actually supported by rsyslog: "Error: Could not find template 0 'RSYSLOG_StdJSONFmt'"

[jonathan-greig]

Closing - the additional formats in the spec don't actually seem to be supported by rsyslog. - See https://github.com/log2timeline/plaso/issues/3012#issuecomment-1350228934