loki

loki copied to clipboard

What this PR does / why we need it:

Allow promtail to skip or stop tailing watching files/folders that contain a file with promtail_ignore in the title. promtail works poorly with transient log files (like airflow dag logs) because it tries to watch an ever growing list of files folders which will likely never change again.

This allows you to stop promtail watching those files/folders on the fly

Which issue(s) this PR fixes: Fixes #

Special notes for your reviewer:

Checklist

• [ ] Reviewed the CONTRIBUTING.md guide (required)
• [ ] Documentation added
• [ ] Tests updated
• [ ] Title matches the required conventional commits format, see here
• [ ] Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• [ ] For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
• [ ] If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

Initial PR to see if there is any views towards it, can work on tests later

All committers have signed the CLA.

sorry @lozbrown but at the moment we're considering promtail to be in a feature complete state and only accepting bug/security fixes

an improvement would be welcome in the sense that if we detected that a file hasn't been updated in a very long time (like in 24h for example, which could apply to systems using something like logrotate) but even then we might have to put that upstream only into grafana alloy

That's really sad to hear, at the moment as an organisation we've been unable to move from splunk to Loki for this use case because promtail continuously ticks up its resource usage until it's utilising several CPU cores flat out.

Basing the dropping of files on the update time potentially makes it difficult to handle files that only move daily or backload logs that have been missed. This solution allows more flexible control for the owner to decide when to stop watching.

Deciding that the product is perfect and needs no new features seems short sighted, please reconsider...

@lozbrown I'm sure the Alloy project would accept your change/improvement here. Their code, other than the configuration format, is actually a 1:1 fork of promtail. Their team is dedicated to building a fully featured telemetry agent for all data types, not just logs or metrics. You'll have a better experience long term using their software.

Deciding that the product is perfect and needs no new features seems short sighted, please reconsider...

We're not saying promtail is perfect, we're saying that as a team we're solidifying our focus on Loki itself and that the Grafana Alloy team is in a better position to build a better agent for logs than we are.