elastic
New configurable service.idle_jobs_threshold
Closes https://github.com/elastic/connectors/issues/2298
This PR creates a new configurable field
service.idle_jobs_threshold
which is the amount of time (in seconds) before a sync job is considered 'idle'.
We make this field configurable by grabbing it from the service configurations and passing the value to SyncJobIndex::idle_jobs() as an new argument - previously this was a hardcoded global variable.
TODO
- [ ] Setting this value to zero should disable 'idle' jobs fully (?)
- [ ] Possibly provide a log message as to why this timeout occurred
Checklists
Pre-Review Checklist
- [x] this PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check
config.yml.example) - [x] this PR has a meaningful title
- [x] this PR links to all relevant github issues that it fixes or partially addresses
- [x] if there is no GH issue, please create it. Each PR should have a link to an issue
- [x] this PR has a thorough description
- [x] Covered the changes with automated tests
- [x] Tested the changes locally
- [x] Added a label for each target release version (example:
v7.13.2,v7.14.0,v8.0.0) - [ ] Considered corresponding documentation changes
- [ ] Contributed any configuration settings changes to the configuration reference
There's one thing to keep in mind here - Kibana has its own threshold where it'll also mark jobs as "Idle" in UI. Don't remember where exactly, but somewhere in our module.
Connectors fail the job once idle threshold is reached, while Kibana just signalises that it haven't seen any updates for a bit and job might be stuck - might be worth to re-visit this part too.
