lucashenning
Caching feature
Hey there, thank you for the plugin. Only one thing is missing in plugin to be perfect - is caching. Add please option to cache value (refreshInterval) in case of successful response. Here is already implemented, so you can just copy-paste (https://github.com/angel9484/logstash-filter-lookup)
Hi, thank you for your proposal. I can only point out my opinion on this, where local caching is maybe not desirable.
I.e. within a scaled environment with multiple logstash instances, different caching timeout settings could lead to inconsistent data. Worst case within the timeframe of
For me the best approach here seems to have such caching on a central instance (the REST service itself or a reverse -- microservice -- proxy cache) to achieve consistent event processing. With http keepalive the overhead and delay seems to be minimal.
The lookup filter (if I understand it correctly) loads a complete data set and refreshes this every x.
From my point of view, this is a different approach. As the rest filter queries a REST service and enriches the processed event, without having to load a set of data in advance.
So a "last recently used" cache (https://github.com/SamSaffron/lru_redux) is what it should be, I guess.
Then you go down the road of multi-threading and parallel processing, which drives complexity...
But long story short: Of course with an option to disable the functionality, you are highly welcome to create a Pull Request.
Even 5 sec of caching saves CPU - for example if you have a bunch of records are coming which do the same queries with the same arguments on their processing. I wish I could create a PR, but my Ruby knowledges are too low :) Maybe if someone would add some caching library (or store it in memore), I could add cache with ttl by copying from that library I gave in topic. :)
Hm, well, I wonder how to easily/efficiently identify a http request... It might consist of an url, body or method to build up and use a lru cache.
Hey @gandalfb, basicly request can be counted as unique if it returns the same result. This consists of:
- url
- method
- body
As result by doing two the same request you'll guaranteed receive the same response (if server state, env settings, and other remove stuff keeps the same)
@logical-and cache also can be implemented on serverside ( where the api lives ). @gandalfb Of course every request going to cost a connection which also costs cpu time so a local cache could be so desired and at that point i agree with @logical-and :+1:
