Utility to cleanup the MetaDataStore based on some expiry date [INT-4522]

[spring-operator]

Nagendra siram opened INT-4522 and commented

The Spring idempotent service use INT_METADATA_STORE table to identify duplicates messages. 

https://docs.spring.io/spring-integration/docs/current/reference/html/system-management-chapter.html#metadata-store

In above link it says

The value of the idempotent entry may be some expiration date, after which that entry should be removed from Metadata Store by some scheduled reaper.

It would be nice to have a utility function provided by spring that user can wire up to clean up the records that are in INT_METADATA_STORE table.

---

Reference URL: https://stackoverflow.com/questions/51839414/how-to-cleanup-the-metadata-store-used-for-idempotent-spring-integration-pattern

Referenced from: pull request https://github.com/spring-projects/spring-integration/pull/2619

[spring-operator]

Artem Bilan commented

See related StackOverflow question about more info.

This is really a matter of the MetadataStore and there is no "idempotent service" to avoid readers' confusion.

[spring-operator]

David Turanski commented

I tried to implement something like MessageGroupStoreReaper as a MetadataStoreListener in order to track all the keys but not all the implementations (JDBC) are ListenableMetadataStore. MetadataStore does not expose anything like allKeys(). The original reaper delegates to MessageGroupStore.expireMessageGroups(). So something like this won't work without changes to the MetadataStore. Also Listener has some similar overhead since it is caching all the keys.  If the expiration strategy is reasonable, it will clean itself up at the the same time.  Some thoughts:

• Add a functional interface to MetadataStore to apply a function to all entries (or something more concrete, similar to MessageGroupStore)
• Make all current implementations ListenableMetadataStore
• Add allKeys to MetadataStore

  

[spring-operator]

Artem Bilan commented

The ListenableMetadataStore makese sense only in those MetadataStore where we really have a callback from the target store. For example Hazelcast. In case of JDBC there is no such a push-notification from the RDBMS (not talking that we can implement something ourselves with triggers and some subscriber e.g. Oracle AQ...).

My idea here is something like ExpirableMetadataStore with the contract like:

void expireOlderThan(long age);

So, this way we really need to extend our MetadataStore structure to additional LAST_UPDATE property. I believe this way we could remove old entries with some scheduled task pretty easy.

[spring-operator]

David Turanski commented

ExpirableMetadataStore will work.

LAST_UPDATE is an improvement. By default the value is a timestamp, so I was assuming we can just use that unless we get a NumberFormatException, but I'm guessing ValueStrategy is not used very much so this would work in must cases. In any case, we still need a way for the scheduled task to have access all the entries.  

[spring-operator]

Artem Bilan commented

Yeah... Agree. The Set<K> keySet() would be great addition as well.