Use a better folder structure to organize check configurations

[martinisoft]

This is a feature request. I'm a current customer of the service so I could also go through the support channels if that is more appropriate.

Additional environment details (Operating System, Cloud provider, etc): Running datadog agent 5.14.1 on Ubuntu 14.04.4

Steps to reproduce the issue:

1. Write a check configuration file in conf.d with a name that does not directly correlate to a python check in the checks.d folder

Describe the results you received: The check will not execute due to the current folder structure and naming scheme. Check configurations must match the check names. This breaks the well established .d folder pattern present in Unix SysV based systems.

Describe the results you expected: I would expect to be able to name the configuration files in conf.d anything I'd like because of the already widely established convention of a .d folder that collects fragments of configurations in order to generate the overall configuration for certain software.

Additional information you deem important (e.g. issue happens only occasionally): Switching to the established .d folder convention benefits everyone in the following ways:

1. UX benefits for seasoned sysadmins who expect the conf.d folder to work like any other .d in their system. Less head scratching and probably less support load.
2. Drastically simplified configuration management as each configuration file can be managed separately by the users configuration management software of choice.

This change will hurt because it is a fundamental shift from your established pattern of matching configuration file to check. It will be a breaking change, but this is what major version numbers are for in semantic versioning. I think the long-term benefits outweigh the cost of breaking from a poor design decision.

[masci]

Hi @martinisoft and thanks for your detailed feedback, much appreciated.

I don't think something like this will ever happen for the 5.x release line but we could take this into consideration as a feature request for the upcoming 6.x.

The logic to pick up and schedule a check for running in the new agent is:

1. the agent finds a check configuration (on a file or by other sources)
2. by looking at the configuration, the agent is able to load and execute the Python bits for the corresponding check.

Point 1 can't be changed, the entire Agent is based on that assumption but Point 2 is open for discussion. At the moment, in the case of a configuration living in a file on disk, we still rely on the file name to match the corresponding check (we simply ported the same strategy we have in 5.x) but this can be changed.

Could you provide something like a "user story" or simply an example for your use case?

Thanks!

@irabinovitch might be interested in this discussion.

[martinisoft]

@masci I'd simply like to have the conf.d folder contain any number of .yaml files which are globbed and evaluated together. The consequence I'd see here is forcing all check configurations to namespace their YAML data under the key for their intended check like so:

dns_check:
  init_config: []
  instances: {}

This should allow you to specify any amount of configuration files and they will ultimately merge under the same key. Order should not matter in this case since we're globbing, but if it is needed then putting numbers in front of the filenames to force a sort order will help that if needed.

In a broader example, given a directory like this:

conf.d
- examplecheck.yaml
- customcheck.yaml
- webappcheck.yaml

With examplecheck.yaml containing:

dns_check:
  init_config: []
  instances:
    - name: myhost
       hostname: example.com

And customcheck.yaml containing:

custom_check:
  init_config: []
  instances:
    - name: newcheck
       attribute: foo

With webappcheck.yaml containing:

dns_check:
  init_config: []
  instances:
    - name: webapp
       hostname: mywebapp.com

Results in this computed configuration:

dns_check:
  init_config: []
  instances:
    - name: myhost
       hostname: example.com
    - name: webapp
       hostname: mywebapp.com

custom_check:
  init_config: []
  instances:
    - name: newcheck
       attribute: foo

This executes the dns_check and custom_check plugins for the agent with all the attributes merged. In the form of a more generalized user story:

As a system administrator
I would like to create check configurations with any filename
So that I can organize my check configurations to suit my environment setup

[xkrt]

This is really painful to support configuration file for a check without some kind of ".d" support with any provision tools (we use Ansible).

I think there is the alternative which may be released easily in 5.x. Directory layout of conf.d for @martinisoft example:

./conf.d/dns_check.d/myhost.yaml
./conf.d/dns_check.d/webapp.yaml
./conf.d/customcheck.yaml

With myhost.yaml containing:

init_config:
instances:
  - name: myhost
    hostname: example.com

With webapp.yaml containing:

init_config:
instances:
  - name: webapp
    hostname: mywebapp.com

With customcheck.yaml containing:

init_config:
instances:
  - name: newcheck
    attribute: foo

So algorithm of dd-agent will be:

1. Read all ./conf.d/{check-name}.yaml configs if any
2. Read all ./conf.d/{check-name}.d/{whatever}.yaml configs if any
3. Merge results of 1. and 2.
4. For all readed {check-name}'s run checks with merged configs

@masci what you think about it?

[masci]

@xkrt this is exactly what we do in the new agent: https://github.com/DataDog/datadog-agent/blob/master/pkg/collector/providers/file.go#L120 Not as good as what @martinisoft suggested but this way we can easily provide backwards compatibility.

Not sure I'll have the bandwidth to backport this feature to v5 anytime soon but the idea is definitely good, I will support any contribution effort in this direction.

[xkrt]

@masci what is https://github.com/DataDog/datadog-agent/ ? new version (6.0) of https://github.com/DataDog/dd-agent? why it hosted in new repo? when it will be available for customers?

[irabinovitch]

@xkrt https://github.com/DataDog/datadog-agent/ is a complete rewrite of the agent which will be released as 6.0. Beta1 was released at Datadog Summit last week and is available now.

[masci]

Here the docs with the instructions if you want to give it a shot: https://github.com/DataDog/datadog-agent/blob/master/docs/beta/README.md