rsyslog
Rsyslog stops collecting logs after certain period of time while using omprog module
Expected Behavior: rsyslog to collect logs and send logs to matomo directly from containered rsyslog.
Actual behavior: Containerized rsyslog stops collecting logs after certain period of time on server B. I am using omprog module within rsyslog.conf to execute a script which is sending logs to matomo. Until I restart rsyslog container it doesn't collect any log.
I am using the same rsyslog image on two servers, where the server A (where rsyslog.conf is not using any omprog module to run the script) has no problem in collecting logs but this server has.
Environment: Ubuntu : Ubuntu 20.04.1
rsyslog version:
rsyslogd 8.2001.0 (aka 2020.01) compiled with:
PLATFORM: x86_64-pc-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Config file: /etc/rsyslog.conf
PID file: /run/rsyslogd.pid
Number of Bits in RainerScript integers: 64
Below is my rsyslog.conf.
# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")
# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")
# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
# Collect log with UDP
$ModLoad imudp
$UDPServerAddress 0.0.0.0
$UDPServerRun 8514
# Creating separate log files based on the severity
#local1.* /var/log/haproxy-traffic.log
#local1.notice /var/log/haproxy-admin.log
$ActionFileDefaultTemplate CustomFileFormat
$template CustomFileFormat,"%$!msg:::drop-last-lf%\n"
if $msg contains 'export.serverB.XX.com' then {
set $!msg = ltrim(rtrim(replace($msg, '"export.serverB.XX.com"', '')));
/var/log/export.serverB.XX.com-access_log;CustomFileFormat
} else if $msg contains 'www-serverB.com' then {
set $!msg = ltrim(rtrim(replace($msg, '"www-serverB.com"', '')));
/var/log/www-serverB.com-access_log;CustomFileFormat
} else {
set $!msg = ltrim(rtrim($msg));
/var/log/serverB-haproxy-traffic;CustomFileFormat
}
module(load="omprog")
action(type="omprog"
binary="/opt/matomo/matomo.sh"
template="CustomFileFormat"
confirmMessages="off")
#
The version you are using is a bit old and no longer supported by the community. It is also very probable that the problem stems back to solved issues. As such, please update to the currently supported version and retry. Report back if the problem persists.
The rsyslog project provides packages for current versions: https://www.rsyslog.com/downloads/download-other/
If you insist on using an outdated version, you can also contact your distro or consider rsyslog professional services.
Could it be that the script hangs or responds very slowly? This could push-back rsyslog processing.
Ideally, we would like to have a debug log from when the problem occurs. But I guess this is too seldomly the case. I ask because it is usually impractical to run for hours in debug mode.
Im suffering from the same issue here with same rsyslogd version. Rsyslog stops logging after ≈ 30 minutes. if i restart the service it will work fine for around 30 minutes then it will hang again.
rsyslogd 8.2001.0 (aka 2020.01) compiled with:
PLATFORM: x86_64-pc-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Config file: /etc/rsyslog.conf
PID file: /run/rsyslogd.pid
Number of Bits in RainerScript integers: 64
See https://www.rsyslog.com for more information
The config file looks like the following :
# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")
# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")
# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
