[Feature Request]: An eventually consistent, cached, execution information file location

[mattcasters]

What would you like to happen?

The current file location for execution information is quite granular in the way that it stores information. It does this to avoid too many delays when updating existing information with new pieces of data. I think we can work around this with a simple caching strategy. We can allow the information to simply end up in a memory structure with periodic writes to disk. We could opt to flush entries to disk when they haven't been updated for a few seconds (configurable) and we could remove information from memory after a certain idle time. (pipeline or workflow execution information hasn't been updated for a few minutes, configurable)

Issue Priority

Priority: 3

Issue Component

Component: Other