Dynamic configuration values

[alexander-gorelik]

Ok so let's start from the context

We encounter scenarios when dynamic configuration values are required. I have also seen a lot of questions in slack regarding that.

We want to find some specific instance(by tag) from an instance group, for example, an instance that acts as a master, and tag it with some "under-experiment-tag". Then we are going to use this instance-id for some additional checks, that are executed in different probes(we want our activities to be independent) and in the end, we are going to kill it in action.

The problem here is that for all those activities we need the instance id and as we know CTK doesn't store any state.

So a couple of months ago we came with a solution, better to say that this is a patch, we created a global configuration with the state that is shared between all the activities, acts the same as env variables. That solved our problem temporarily but it caused different code design problems that we didn't like. First of all, get/set value from configuration is nailed into functions and I am highly anti-global configuration as a solution to problems, it's fast but will create hell in the future if used without cause, and that is what usually happens with a lot of contributors to the codebase. In addition, it makes unitests look not so good, and that is a sign that there is a problem with a code.

    """Tag instance.

    Args:
        aws_ec2_client: AwsEc2
        experiment_tag: tag which should be added for chosen instance
    """
    instance_id = ExperimentGlobals().config.get("instance_id")
    aws_ec2_client.tag_instance(instance_id, experiment_tag)

So we rethink this problem today and came up with a solution that can be implemented in two ways

1. In CTK Infra by enhancing the load_configuration method to support not only static and env but also the python. Here is the example of config.

"configuration": {
	  "Instanceid": {
	    "type": "python",
	    "module": "module",
	    "func": "get_instance_id",
	    "arguments": "whatever"
	  }
}

As a result during config, the function will be called and the value that it returns will be assigned to the configuration key, so the configuration will look like

"Instanceid":"some_instance_id"

In that way, this key can be passed as an argument with normal "${}" CTK pattern That is the tidy solution that will enhance CTK

2. Custom solution - almost the same code as the first one We will use the same configuration described in the first step, in the controls, before configuration is loaded we will scan the raw exp.json file to find the pattern of this configuration, execute the function and override the value.

Of course, we can help implement that if it makes sence. What do you think?

[WixoLeo]

Arguments is an object obviously

"configuration": {
	  "Instanceid": {
	       "type": "python",
	       "module": "module",
	       "func": "get_instance_id",
	       "arguments": {
                     "my_arg": 2
                }
	  }
}

[Lawouach]

Hey guys,

I think this one time where both controls and extending configuration could be applied. I like the idea of using python in the config file as you describe.

However, I'm still a bit confused about where this instance id comes from? I mean I understand we would call a Python function to retrieve it but what does this function reference to find it?

[alexander-gorelik]

The instance id is just an example, it could be anything that you need in order to pass to your multiple activities.

In the scenario, I described we get a list of instances from aws by tag(that will be passed as an argument), then select a random instance from that list, and return the instance id.

It could be used to tag instance with a chaos tag in one action (the example above), perform kill in other action, recover in rollback, and others...

[Lawouach]

Right, so i'm happy to have a new config privider added that uses Python to load a function that returns the expected value. IMO, the same validation should occur as on actions/probes: checking the module/function exist and can be imported, checking the signature to see if they match the passed arguments.

[je-al]

FWIW, controls do work, i.e.: assigning a value to the configuration instance passed as an argument from an after_activity_control is persisted and can be used as arguments for other activities/probes.

It's trying to make a generic one that gets a bit complicated, my current implementation receives a list of names of activities which to operate on:

extensions:
  - name: my-ext
    data: |
        - 'cat':
          - name: 'service_requirements'
            value: '$.service_metadata'

(using a yaml formatted string just because it happens to be highlighted and reads nicely)

import yaml
from jsonpath_ng.ext import parse

_config = {}
_extension_name = 'my-ext'


def configure_control(configuration: Configuration = None,
                      secrets: Secrets = None, settings: Settings = None,
                      experiment: Experiment = None):
    global _config
    _config = {
        e['name']: yaml.safe_load(e['data'])
        for e in experiment['extensions']
        if e['name'] == _extension_name
    }


def after_activity_control(context: Activity, state: Run,
                           configuration: Configuration = None,
                           secrets: Secrets = None, **kwargs):
    for extraction in _config.get(_extension_name, {}).get('extract', []):
        for name, value in extraction.items():
            if name == context['name']:
                for rule in value:
                    matches = [match.value for match in parse(rule['value']).find(state['output'])]
                    if matches:
                        if rule.get('multi', False):
                            configuration[rule['name']] = matches
                        else:
                            configuration[rule['name']] = matches[0]

The code gets more complex when trying to deal with modules whose output is nested, like process (nested yaml parsing), also, it won't work if you actually need more than one separate value from the output, since there's no way to substitute nested values, e.g.: [2022-02-02 22:53:42 DEBUG] [activity:205] => succeeded with '{'status': 1, 'stdout': '', 'stderr': '/bin/sh: ${service_requirements.service_metadata}: bad substitution\n'}'

[Lawouach]

Hey @je-al

I appreciate the code sample but I'm still unsure what problem needs solving precisely here.

[je-al]

Roundabout way of saying +1. We had the same issue as the OP, I was just providing a sample implementation that works by extracting information from existing probes / activities that then can reused into existing activities (since it puts the state into configuration) for any lurkers...

But also, chipping in that a way to map more than one value from the response might be needed. Either by specifying several mappings into the configuration variables to set or more complex configuration variable substitution.

[Lawouach]

Hey all,

I'm still not clear how to go on about this one. µI can see the appetite but as demonstrated, solutions tend to grow quickly complicated. I think there is always a balance between capability and usability. This time, it's not a balance that is straightforward.

[roeiK-wix]

This issue is done

[github-actions[bot]]

This Issue has not been active in 365 days. To re-activate this Issue, remove the Stale label or comment on it. If not re-activated, this Issue will be closed in 7 days.

[github-actions[bot]]

This Issue was closed because it was not reactivated after 7 days of being marked Stale.