Tailnmail analyzed old logs after server down for some days

[gp4git]

Hello, for maintenance reasons, our PG linux server was down from Friday October 27 to Monday October 30.

In our postgresql.conf we use : log_rotation_age = 1d In tail-nmail config we have : FILE1: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces

So after one year activity, we have 31 PG log files in our LOG directory (from <CLUSTER_NAME>-01.log to <CLUSTER_NAME>-31.log) on which PG rotates each night.

When the server went up, this morning (October 30th), the first run of Tailnmail analyzed files <CLUSTER_NAME>-27 to <CLUSTER_NAME>-30 considering they were all from October but due to the weekend shutdown files 28 and 29 were in fact files from last September. Only files -27 and -30 files should have been analyzed.

Here is the debug output of this tnm run : Opened config file "/u02/pgsql/admin/9.4/<CLUSTER_NAME>/scripts/tnm_pgsql.cfg" $opt1 = { 'configfile' => { 'log_line_prefix' => 1, 'include' => 1, 'file.HASH(0x1270388)' => 1, 'file' => 1, 'email' => 1, '[email protected]' => 1, 'exclude.checkpoint complete:' => 1, 'include.FATAL:' => 1, 'exclude.automatic analyze' => 1, 'include.LOG:' => 1, 'exclude.checkpoint starting:' => 1, 'offset' => 1, 'include.ERROR:' => 1, 'lastfile' => 1, 'customsubject' => 1, 'include.PANIC:' => 1, 'exclude.(execute pdo_stmt|statement: (BEGIN|COMMIT|COPY|DEALLOCATE|LOCK TABLE .ACCESS SHARE MODE|SELECT set_limit|(WITH|SELECT) .(cre_struc|del_pvc|pg_|seq)|SET |$))' => 1, 'mailsubject' => 1, 'exclude.automatic vacuum' => 1, 'exclude' => 1 }, 'lastfile' => { '1' => '/u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log' }, 'log_line_prefix' => ''user=%u,db=%d,'', 'include' => [ 'PANIC:', 'FATAL:', 'LOG:', 'ERROR:' ], 'file' => [ { 'lastfile' => '/u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log', 'original' => '/u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-%d.log', 'suffix' => '1', 'name' => '/u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log', 'offset' => '5870628' } ], 'customsubject' => 1, 'email' => [ '[email protected]' ], 'mailsubject' => 'HOST (Serveur de PRODUCTION <CLUSTER_NAME>) Postgres errors UNIQUE : NUMBER', 'offset' => { '1' => '5870628' }, 'exclude' => [ 'checkpoint starting:', 'checkpoint complete:', 'automatic analyze', 'automatic vacuum', '(execute pdo_stmt|statement: (BEGIN|COMMIT|COPY|DEALLOCATE|LOCK TABLE .ACCESS SHARE MODE|SELECT set_limit|(WITH|SELECT) .(cre_struc|del_pvc|pg|seq)|SET |$))' ] }; $arg1 = { 'verbose' => 1, 'log_line_prefix' => ''user=%u,db=%d,'', 'mailserver' => 'example.com', 'find_line_number' => 1, 'file' => [], 'sortby' => 'count', 'maxsize' => 80000000, 'mailuser' => 'example', 'tsepnosub' => 0, 'debug' => 1, 'duration_limit' => 0, 'sqlstate' => 0, 'dryrun' => 0, 'timewarp' => 0, 'pgmode' => 1, 'nomail' => 0, 'duration' => -1, 'statement_size' => 3000, 'mailport' => 465, 'tsep' => undef, 'flatten' => 1, 'type' => 'normal', 'skip_non_parsed' => 1, 'smtp' => 0, 'tempfile_limit' => 0, 'tempfile' => -1, 'hideflatten' => 1, 'mailcom' => '/usr/sbin/sendmail', 'maxemailsize' => 10000000, 'reset' => 0, 'pretty_query' => 1, 'help' => 0, 'mailmode' => 'sendmail', 'mailpass' => 'example', 'offset' => -1, 'quiet' => 0, 'showonly' => 0, 'version' => 0, 'mailsig' => [], 'mailzero' => 0, 'rewind' => 0, 'pglog' => 'pg' }; Log line prefix regex: (?-xism:^(()()user=[[\w]],db=[[\w]],)(.*)) Log line prefix regex2: (?-xism:^user=,db=,) Log line prefix regex3: (?-xism:^user=) Parsing file: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log File: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log Offset: 5,870,628 Size: 5,871,159 Maxsize: 80,000,000 Adding exclusion: checkpoint starting: Adding exclusion: checkpoint complete: Adding exclusion: automatic analyze Adding exclusion: automatic vacuum Adding exclusion: (execute pdo_stmt|statement: (BEGIN|COMMIT|COPY|DEALLOCATE|LOCK TABLE .ACCESS SHARE MODE|SELECT set_limit|(WITH|SELECT) .(cre_struc|del_pvc|pg|seq)|SET |$)) Exclusion: (?-xism:checkpoint starting:)|(?-xism:checkpoint complete:)|(?-xism:automatic analyze)|(?-xism:automatic vacuum)|(?-xism:(execute pdo_stmt|statement: (BEGIN|COMMIT|COPY|DEALLOCATE|LOCK TABLE .ACCESS SHARE MODE|SELECT set_limit|(WITH|SELECT) .(cre_struc|del_pvc|pg|_seq)|SET |$))) Adding inclusion: PANIC: Adding inclusion: FATAL: Adding inclusion: LOG: Adding inclusion: ERROR: Inclusion: (?-xism:PANIC:)|(?-xism:FATAL:)|(?-xism:LOG:)|(?-xism:ERROR:) MATCH at line 44725 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log MATCH at line 44726 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log MATCH at line 44727 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log MATCH at line 44728 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log MATCH at line 44729 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log Lines found in /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log: 5 Checking for file /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log (last was /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log) Checking for file /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-29.log (last was /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log) Checking for file /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log (last was /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-27.log) Parsing file: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log File: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log Offset: 0 Size: 11,310,774 Maxsize: 80,000,000 MATCH at line 6768 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log MATCH at line 6769 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log MATCH at line 6770 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log MATCH at line 6771 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log MATCH at line 7363 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log MATCH at line 8170 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log MATCH at line 8344 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log MATCH at line 21717 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log Lines found in /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-28.log: 67 Parsing file: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-29.log File: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-29.log Offset: 0 Size: 6,128,680 Maxsize: 80,000,000 No new lines found in file /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-29.log Parsing file: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log File: /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log Offset: 0 Size: 554,364 Maxsize: 80,000,000 MATCH at line 955 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 956 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 957 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 958 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 960 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 961 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 995 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 1039 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 1123 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 1175 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 1941 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 2043 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 2098 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 2166 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 2300 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 2443 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 2701 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log MATCH at line 2936 of /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log Lines found in /u02/pgsql/admin/9.4/<CLUSTER_NAME>/traces/<CLUSTER_NAME>-30.log: 18 Sending mail to: [email protected] Performing final cleanup Saving new config file

Regards