ApacheCommonsFileTailingMessageProducer remember state when restarting [INT-3743]

[spring-operator]

Benedikt Linse opened INT-3743 and commented

The FileTailingMessageProducer (both the ApacheCommonsFileTailingMessageProducer and the OSDelegatingFileTailingMessageProducer) currently do not save their state to disk. Therefore the information about the number of lines of the file that were processed is lost when the JVM exits.

A requirement many organizations have is that a tailer can be restarted and upon restart continues tailing the file at the line number that it was interrupted at. Therefore the line numbers must be saved to a db or a disk.

The following information is from Gary's answer to my question on SO and describes how this should be implemented using the MetadataStore:

Spring Integration has an an abstraction MetadataStore - it's a simple key/value abstraction so would be perfect for this use case.

There are several implementations. The PropertiesPersistingMetadataStore persists to a properties file and, by default, only persists on an ApplicationContext close() (destroy()).

It implements Flushable so it can be flush()ed more often.

The other implementations (Redis, MongoDB, Gemfire) don't need flushing because the data is written immediately.

---

Reference URL: http://stackoverflow.com/questions/30774430/spring-integration-filetailingmessageproducer-remember-current-line-when-restar

1 votes, 2 watchers

[spring-operator]

Gary Russell commented

Bear in mind, though, that this solution is not perfect in that the file could roll over between executions so you could miss some data.

But, I agree this would be a useful enhancement to keep track of the position in the current file so, if it's the same file on the next execution, we can skip those lines already emitted.

[spring-operator]

Gary Russell commented

This is non-trivial - moving to the 4.3 backlog.

Issues:

• file rollovers - resetting position in the new file
• detecting file changes between invocations - what to do - error? continue?
• it probably needs to be aware of the rollover strategy so it can catch up on "missed" files.