Filebeat - Provide file's last modification date

[ogauchard]

Hi,

For our use case, we need to use the ingested file's last modification date. But there is actually no way to retrive it using filebeat.

Our use case : we are indexing whole files (one file = one event), and we don't have no date in the file content. We don't want to use the indexing timestamp because we are also indexing "old" files and the indexing timestamp will not match the real file's date.

After asking for a solution on the forum, @ruflin suggested to open a feature request as it can be quite easily done in filebeat. The question is about the data overhead for users who do not need this information.

[ruflin]

As your use case is not the typical logging use case where the mod date would be constantly changing, I could also think of a solution which involves creating a new prospector type for sending complete files. The reason I'm thinking of a new prospector is that the rotating log case is rather complex compared to ingest static files line by line. There are quite a few poeple using it like this, means we could simplify the state handling in such a new prospector. And in such an environment the mod date also makes more sense.

[ogauchard]

I totally agree with you. Ingesting the whole file to create only one event is a diversion of the primary goal of the actual log prospector. And it implies dealing with multiline option which I think is not a good way of doing it. I approve the creation of a dedicated prospector (and hope it is not a big task !)

[rctlopes]

This would be a great feature for one of my cases. I'm reading logs from a 3rd party system (SimCorp), and each line of the log files only have the time, not the date. Receiving the last modified date would bring the missing piece to the logs, and we could create proper record date for the log in Logstash.

[Charly66]

Hi, this enhancement will be great for Talend's job's logging as the timestamp isn't mentionned in logs.

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rctlopes]

Any update on this feature? It would be very helpful having it implemented. Thanks!

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[rctlopes]

Is there any update on this? Thanks!

[cmacknz]

Sorry, we still haven't prioritized this.

[nimarezainia]

@rctlopes at the moment we don't have a plan to address this issue due to many other product priorities that we have.

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[rctlopes]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

👍🏻