Ability to exclude file formats from harvesting

[madhusudan-d]

log-courier shipping all files in configured directory along with .gz(zipped files). Is there any way with which we can tell log-courier not to include specific file formats??. There is exclude option in logstash for doing this. do we have any similar configuration in log-courier??. I am using ELK stack and I can see some garbage data from .gz (zipped) files.

Thanks in advance.

[driskell]

What is your "paths" set to?

If you include .gz files it will try to ingest them. There is usually no need for exclude in this case as you can just include "access-*.log" for example in order to not include the gz files.

Also, in nearly all cases you need only follow the current file, so can just specify "access.log" - it will follow rotations as necessary.

If you're ingesting a special set of logs with unique rotation handling then let me know and we can work something out!

[madhusudan-d]

The path is set to /service-abc/* . and srervice-abc have many log files which are of different formats. we have few of .log and few of _log and few other formats.

[driskell]

You can specify multiple paths as such:

"paths": [ "/service-abc/*.log", "/service-abc/*_log" ]

Would that work?

[driskell]

I will consider adding an "exclude_paths" option onto the roadmap for version 2. For now though hopefully the above will help?

[madhusudan-d]

sure.. Thanks....

[driskell]

Not had any other request for this and not needed it myself so will close this, but happy to receive PRs.