Programmatic log_level change

[seanstory]

Problem Description

Support has noted that right now, changing log_level for Cloud deployments can be tricky, as a change to log_level has a prerequisite for the cloud user to first enable log delivery - even if it's only for support that we want to increase the log verbosity.

While we think this is a mistake and should not be a prerequisite for us to provide support, we do have the ability to help work around this prerequisite while the support team engages the cloud team to address that behavior.

On our end, we could look at making log_level something that can be changed dynamically, without a file edit and restart.

In fact, that's how our log level is set anyway. See:

• https://github.com/elastic/connectors/blob/5c6446804af4d27010130eb74340ec608b8a7f64/connectors/service_cli.py#L188-L192

Proposed Solution

We have some options.

• we could add log_level to the index metadata of .elastic-connectors, along with our default values for ingest pipeline settings. This wouldn't technically require a change to the protocol, but would eventually require a Connectors API when the connectors index becomes a system index.
• we could add log_level to the connector protocol, and store it as a field in .elastic-connectors. This would require API changes now. It would allow each connector to run on its own log level.
• we could add log_level to the connector protocol, and store it as a field in .elastic-connectors-sync-jobs. This would require API changes now. It would allow individual jobs to run on their own log level.

With any of these, we could add UIs, or not, to configure log level.

Alternatives

• we can leave log level changes as-is, requiring a file edit to config.yml and a service restart