Big log files are collected as insights-client(3.x) collect many duplicated lines

[chenlizhong]

Below is an example. The fatal message is printed 3 lines each second. For 1 hour, 3 * 60 * 60 = 10,800 lines. Now insights-client collect all these lines.

Expected result: Just return latest lines to reduce the file size.

postgresql log example:

2019-04-07 23:59:38 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:38 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:38 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:39 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:39 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:39 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:40 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:40 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:40 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:41 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:41 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:41 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:42 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:42 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:42 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:43 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:43 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:43 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:44 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:44 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:44 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:45 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:45 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:45 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:46 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:46 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:46 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:47 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:47 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections
2019-04-07 23:59:47 CDT FATAL:  remaining connection slots are reserved for non-replication superuser connections