[RFC] Dynamic Configuration Service

[huyaboo]

Proposal

Currently, all configs are statically determined via the config schema defaultValues property and the opensearch_dashboards.yml. This provides a centralized repository of config values but a major limitation is the inability to change configs without a server restart. For instance, if the admin needs to set the server.maxPayloadBytes, they would need to go into the config file, edit it, and restart Dashboards. Additionally, the existing config service does not allow for per-request config values. Say an admin needs to set different CSP rules based on the request. This is not feasible with the current model. The proposal instead is for a new core service dynamicConfigService, an extension of the existing configService, to expose a read-only dynamic config client in the RequestHandlerContext for plugins to consume and obtain configs from an external config store. For core services, the client will be exposed in the dynamicConfigService.start() method.

Background

Currently, the configs are handled by the ConfigService in @osd-config, which takes the defaultValues defined in a plugin/core service config schema and overrides them with the config values found in opensearch_dashboards.yml. Take csp configs for example. Its config schema is the following:

export const config = {  
  path: 'csp',  
  schema: schema.object({  
    rules: schema.arrayOf(schema.string(), {  
      defaultValue: [  
        `script-src 'unsafe-eval' 'self'`,  
        `worker-src blob: 'self'`,  
        `style-src 'unsafe-inline' 'self'`,  
      ],  
    }),  
    strict: schema.boolean({ defaultValue: false }),  
    warnLegacyBrowsers: schema.boolean({ defaultValue: true }),  
  }),  
};

This means that by default, the csp config will be the following:

const DEFAULT_CSP = {
	rules: [`script-src 'unsafe-eval' 'self'`, `worker-src blob: 'self'`, `style-src 'unsafe-inline' 'self'`],
	strict: false,
	warnLegacyBrowsers: true,
};

If we define the following config in the opensearch_dashboards.yml file:

csp.strict: true
csp.rules: ["default-src 'self'"]

The final config exposed to all core services and plugins will be

const DEFAULT_CSP = {
	rules: [`default-src 'self'`],
	strict: true,
	warnLegacyBrowsers: true,
};

For plugins, the config schema can be found in config.ts.

Terminology

As to not introduce confusion later in the doc, the below terms will be used

configService

This is the existing OSD config service, which can be found at @osd-config.

dynamicConfigService

This is an extension of the configService, which enables plugins/core services to define config values in a config store and fetch dynamic values for a subset of configs.

config store

The external store which contains all configurable OSD configs. The default store will be an OpenSearch index .dynamic_opensearch_dashboards

config schema

This is the schema type object used to define the structure, value types, and default values of a core service/plugin config

• For plugins: this is found in the config.ts. Take home plugin for example; its config schema is the following

export const configSchema = schema.object({
	disableWelcomeScreen: schema.boolean({ defaultValue: false }),
	disableNewThemeModal: schema.boolean({ defaultValue: false }),
});

config object

This is a generic object which contains all the values of a given configPath. This is the return value for the dynamicConfigService client getConfig

// Example config object for 'home' plugin
const homePluginConfig = {
	disableWelcomeScreen: false,
	disableNewThemeModal: false,
};

configPath

This is the key value that uniquely identifies configs for all core services and plugins

• For core services: this is part of the path field of the config schema.
  • example: the HTTP config configPath is server, which is found in the config schema
• For plugins: this is the configPath found in the plugin manifest file (opensearch_dashboards.json). If this is not specified in the manifest, the plugin id should be used

async local storage

Async local storage are Node provided classes that enable necessary request context to be exposed throughout a request lifecycle. This is needed to call the dynamicConfigService client since different requests can fetch different config values.

Out of Scope

Creating a management page

To access/update config values from the Dashboards UI, a page should be created. However, the specifics of such a page are outside the scope of this RFC.

Determining which configs can be dynamic

This RFC seeks to establish a framework for setting configs as dynamic. A general guideline for what constitutes a dynamic config is as follows:

• Configs resolved during setup() and cannot be modified are not dynamic
• A config cannot direct affect the plugin dependency graph.
• A plugin cannot be enabled/disabled dynamically. This is because some plugins have hard dependencies on other plugins and functionality assumes the plugin dependency tree is set once. If the enablement can be modified, unexpected behavior can arise.
• OpenSearchService configs cannot be made dynamic as the earliest time the DynamicConfigService can start is after OpenSearchService finishes starting (so that the OpenSearch client can be created)

Plugin owners and core service owners should determine which of their configs can be made dynamic.

High-Level Approach

As stated in the Proposal, there should be a new core service DynamicConfigService, which is an extension of the existing ConfigService. The configs taken from the DynamicConfigService will override the ConfigService configs. In essence, what this means is there is a priority on the config values:

1. If a config value is set in the config store, the value will be used
2. If this value cannot be found in the config store, the value set in the opensearch_dashboards.yml will be used
3. If this value was not set in opensearch_dashboards.yml, the config schema defaultValue will be used

On initialization/before setup()

The DynamicConfigService instance will need to follow ConfigService initialization in order to consume its public methods. It should also be exposed to CoreContext so other core services can consume the service easily.

// This can be found in src/core/server/server.ts

this.logger = this.loggingSystem.asLoggerFactory();  
this.log = this.logger.get('server');  
this.configService = new ConfigService(rawConfigProvider, env, this.logger);  

// Will take configService as a dependency
this.dynamicConfigService = new DynamicConfigService(this.configService, env, this.logger);  

const core = {  
  coreId,  
  configService: this.configService,  
  dynamicConfigService: this.dynamicConfigService,  
  env,  
  logger: this.logger,  
};

In particular, the DynamicConfigService also requires plugins and core services to have their schemas registered by calling setSchema() to leverage existing schema validation logic. While ConfigService already does this, its schema map is private.

// Registering core service config schemas (src/core/server/server.ts)
for (const descriptor of configDescriptors) {  
  if (descriptor.deprecations) {  
    this.configService.addDeprecationProvider(descriptor.path, descriptor.deprecations);  
  }  
  await this.configService.setSchema(descriptor.path, descriptor.schema);  
  this.dynamicConfigService.setSchema(descriptor.path, descriptor.schema);  
}

// Registering plugin config schemas (src/core/server/plugins/plugins_service.ts)
const configDescriptor = plugin.getConfigDescriptor();  
if (configDescriptor) {  
  ...
  await this.coreContext.configService.setSchema(  
    plugin.configPath,  
    configDescriptor.schema  
  );  
  this.coreContext.dynamicConfigService.setSchema(  
    plugin.configPath,  
    configDescriptor.schema  
  );  
}
...

setup()

The DynamicConfigService setup should happen right after the ConfigService sets up. There are several reasons why:

1. ConfigService on setup() will discover all plugin configs and construct a plugin dependency graph. To avoid duplicate efforts, the DynamicConfigService should not concern itself with this
2. ConfigService performs a validation step in its setup() that ensures config schemas are well formed, have unique namespaces, and opensearch_dashboards.yml values are valid and are valid configPaths

When the setup() finishes, the DynamicConfigService will expose the following methods:

export interface InternalDynamicConfigServiceSetup {
	/**
	* Enables other plugins/core services to register their own config store client. By default,
	* an OpenSearch client is provided, but a plugin can register a config store of their choice
	* like PostgreSQL, DDB, MongoDB, etc...
	* 
	*/
	registerDynamicConfigClientFactory: (factory: IDynamicConfigStoreClientFactory) => void; 
	/**
	* Enables other plugins/core services to register header keys to be captured by the async local store.
	* For example, registering 'request-id' will make this key accessible to the DynamicConfigClient
	* 
	*/
	registerAsyncLocalStoreRequestHeader: (key: string | string[]) => void;
	/**
	* For core services ONLY. This function will return a promise in the case a core service needs the client
	* during the setup() stage.
	* 
	*/
	getStartService: () => Promise<InternalDynamicConfigServiceStart>;
}

export type DynamicConfigServiceSetup = Pick<InternalDynamicConfigServiceSetup, 'registerDynamicConfigClientFactory' | 'registerAsyncLocalStoreRequestHeader'>;

Between setup() and start()

To make configs configurable to the admin, CRUDL routes have to be registered. However, if the HTTP route registration is handled in setup() stage, there will be a circular dependency since httpServer now depends on the dynamicConfigService and the dynamicConfigService depends on httpService. As a workaround, the dynamicConfigService will have an intermediate step registerRoutesAndHandlers() (tentatively named) to register http routes and other handlers after http finishes setting up. This serves two purposes: to register http routes and the async local store in the preAuth server extension. This is what enables a per request context to be feasible:

public async registerRoutesAndHandlers(setupDeps: RegisterHTTPSetupDeps) {  
	const { http } = setupDeps;

	registerRoutes({
		http: http,
		...
	});

	this.#logger.info('registering middleware');  
	// Register the async local storage with all the registered localStorage headers
	http.server.ext('onPreAuth', (request, h) => {  
		const localStore = createLocalStore(this.#logger, request, this.#requestHeaders);  
		this.#asyncLocalStorage.enterWith(localStore);  
		return h.continue;  
	});  
}

start()

In start(), the service will create the readonly client for other core services and plugins to consume. If a custom client factory was provided, the custom client will be created. Otherwise, the default OpenSearch client will be created. When start(), finishes the service will expose the following methods:

export interface InternalDynamicConfigServiceStart {  
	/**
	* For core services ONLY. This will expose a read-only client to read config values
	*/
	getClient: () => IDynamicConfigurationClient;  
	/**
	* For core services ONLY. This will expose a read-only async local store map to pass into the client
	*/
	getAsyncLocalStore: () => AsyncLocalStorageContext | undefined;  
}

// No DynamicConfigServiceStart will be exposed to plugins

DynamicConfigService should be the first service to start so that other core services can use its client.

Example usage

For plugins

// Getting the asyncLocalStore
const store = context.core.dynamicConfig.asyncLocalStore;

// Getting the dynamic config client
const client = context.core.dynamicConfig.client;

// For plugins with a config path in its manifest file
const configObject = await client.getConfig({ configPath: "INSERT-PLUGIN-CONFIG-PATH" }, { asyncLocalStorageContext: store!});

// For core services/plugins without a config path in its manifest file
const configObject = await client.getConfig({ name: "INSERT-PLUGIN-ID-OR-CORE-SERVICE" }, { asyncLocalStorageContext: store!});

// Accessing the value
const bar = configObject.foo;

For core services

// Getting the asyncLocalStore
const store = internalDynamicConfigServiceStart.getAsyncLocalStore();

// Getting the dynamic config client
const client = internalDynamicConfigServiceStart.getClient();

// For plugins with a config path in its manifest file
const configObject = await client.getConfig({ configPath: "INSERT-PLUGIN-CONFIG-PATH" }, { asyncLocalStorageContext: store!});

// For core services/plugins without a config path in its manifest file
const configObject = await client.getConfig({ name: "INSERT-PLUGIN-ID-OR-CORE-SERVICE" }, { asyncLocalStorageContext: store!});

// Accessing the value
const bar = configObject.foo;

Challenges

Scoping the client for plugins

Currently, all configs are isolated per plugin. This means that plugin A cannot access plugin B without B explicitly exposing its configs. How ConfigService handled config object scoping was leveraging the PluginInitializerContext to expose a create function that would create the scoped down client. Since plugins can register their own custom config store client in the setup() stage and the client can only be used in the request context, the initializer context cannot be used to create the scoped down client.

Core service limitations

While core services can have configs, they cannot be disabled meaning DynamicConfigService will exist as a permanent service. This can cause excess network calls to OpenSearch if a Dashboards instance does not use this feature. A workaround could be to define some "dummy" client that only returns the configs from the ConfigService and set that as the client if the service is "disabled".

Alternatives

A plugin was initially considered to make the service easy to enable/disable and prevent an extensive core change. However, plugins cannot access all configs because configs are isolated per plugin. Additionally, core services cannot be changed as core services cannot depend on a plugin (circular dependency).