Support loading parameters from different data sources

[hongchaodeng]

Currently, writing a Configuration yaml requires filling in all parameters one by one. It would be convenient if we can load the parameter values from data sources automatically. For example, a ConfigMap have application data:

apiVersion: v1
kind: ConfigMap
metadata:
  name: server-env
data:
  SERVER_DOMAIN: internal.my-company.com
  SERVER_PORT: "80"

apiVersion: core.hydra.io/v1alpha1
kind: Configuration
spec:
  parametersFrom:
    configMapRef: server-env

Secondly, we want to load it from different sources like k8s ConfigMap, etcd.

[technosophos]

Again, this raises a question of how we would do this for non-Kubernetes implementations. Is it safe to assume that such platforms have any notion of a persistent external configuration?

[wonderflow]

Hydra may want to give such extend ability to give user this choices from which source there parameters were stored.

[wonderflow]

I find this pr is similar to https://github.com/microsoft/hydra-spec/issues/14 and https://github.com/microsoft/hydra-spec/issues/29

[resouer]

For now I feel it's difficult to make such assertion as it actually involves two assumptions:

1. Data source: a system readable persistent configuration source
2. Data format: the key-value style config data which could be converted to parameters filed

[wonderflow]

As I said in #14

For example, I have two components, one is zookeeper and the other one is kafka. I have a workflow to orchestrate these two component, so this will install zookeeper first and pass zookeeper host as a parameter to kafka, this is dynamic and can't be pre-defined in spec.

My zookeeper and kafka are all workloads and don't have container spec. without this I have no way to solve my case.

[resouer]

@wonderflow Besides the specific use case you mentioned, a more general user story in my mind is Hydra workload whose parameters come from runtime (i.e. not pre-defined).

[resouer]

The same request raised in #251.

We may want to consider a generalized Configuration in OAM which is used for describe configuration data for Dev.

Note that essentially Configuration is not different from embedded configFile.value except it provides:

1. a separate place to keep Component human readable.
2. an interface to integrate configuration source storages in runtimes.

/cc @technosophos @vturecek