elasticsearch
elasticsearch copied to clipboard
elastic
Operator/ingest
This PR adds support for /_ingest/pipeline for file based settings.
Relates to https://github.com/elastic/elasticsearch/issues/89183
I left a few comments, I'm unsure about the preTransform piece. I understand we need the ingest infos, but I wonder if there is a better way to track it somehow so that validation could not require it. What do you think?
Yeah good point, let me go back and look at ways we can do without it. It complicates things quite a bit, for such a small thing. I'll address the rest of the feedback based on what I manage to do here.
I pushed an update to see if all tests pass, this seems like a simpler change and more efficient, since we don't expect node configuration to change that much along with file based settings. We keep the node infos cached in the file settings service, and we only refresh them when some nodes have changed.
I need to write extra tests for that logic, make sure we set the flag correctly when nodes are added and removed. I initially thought I only need to track the adds, but removes are needed in case the last ingest capable node is removed.
I reworked how the node infos were being called, it's done on demand and only if nodes have joined/left the cluster. I also just pushed an update with new tests that ensure the nodeInfos are fetched correctly and at the right time.