`ConfigProvider.fromFileSystem`

[vecerek]

What is the problem this feature would solve?

It is a common security practice to read secrets from the file system (or a memory-based volume pretending to be a file system), instead of the globally available environment variables.

What is the feature you are proposing to solve the problem?

I have implemented a fromFileSystem ConfigProvider. I'm looking for ways to upstream it now. This is how it's implemented currently (very much tied to the Node platform):

import {
  Boolean,
  ConfigError,
  ConfigProvider,
  Effect,
  Either,
  HashSet,
  pipe,
  ReadonlyArray,
  String,
} from "effect";
import { flow } from "effect/Function";
import * as fs from "node:fs";
import * as path from "node:path";

type Parse<A> = (content: string) => Either.Either<A, ConfigError.ConfigError>;

const parseConfig = <A>(parse: Parse<A>) =>
  flow(
    String.trim,
    parse,
    Either.match({
      onLeft: Effect.fail,
      onRight: flow(ReadonlyArray.of, Effect.succeed),
    })
  );

const readConfig = <A>(filePath: string, parse: Parse<A>) =>
  pipe(
    Effect.sync(() => fs.readFileSync(filePath, "utf-8")),
    Effect.flatMap(parseConfig(parse))
  );

const resolveEnumerableDirs = (pathSegments: readonly string[]) =>
  pipe(
    pathSegments,
    ReadonlyArray.isEmptyReadonlyArray,
    Boolean.match({
      onFalse: () => [path.join(...pathSegments)],
      onTrue: () => [],
    })
  );

export const fromFileSystem = (opts?: { rootPath: string }) => {
  const resolveFilePath = (pathSegments: readonly string[]) =>
    path.join(...[opts?.rootPath ?? "/", ...pathSegments]);

  const pathNotFoundError = (pathSegments: readonly string[]) =>
    ConfigError.MissingData(
      [...pathSegments],
      `Path ${resolveFilePath(pathSegments)} not found`
    );

  const listFiles = (pathSegments: readonly string[]) => (dir: string) =>
    Effect.try({
      try: () => fs.readdirSync(dir),
      catch: () => pathNotFoundError(pathSegments),
    });

  return pipe(
    ConfigProvider.makeFlat({
      load: (pathSegments, config) => {
        const filePath = resolveFilePath(pathSegments);

        return pipe(
          fs.existsSync(filePath),
          Boolean.match({
            onFalse: () => Effect.fail(pathNotFoundError(pathSegments)),
            onTrue: () => readConfig(filePath, config.parse),
          })
        );
      },
      enumerateChildren: (pathSegments) =>
        pipe(
          pathSegments,
          resolveEnumerableDirs,
          ReadonlyArray.map(listFiles(pathSegments)),
          Effect.all,
          Effect.map(flow(ReadonlyArray.flatten, HashSet.fromIterable))
        ),
      patch: {
        _tag: "Empty" as const,
      },
    }),
    ConfigProvider.fromFlat
  );
};

The implementation is battle-tested, has been running in production for a while now and can generally read config from any file.

This file system provider has a single config option: rootPath (default: "/") which acts as the filePath prefix for all configuration (comes in handy when testing stuff locally, most operating systems don't give the admin user write permission to the / path). Let's assume, you have a secret stored in /secrets/MYSQL_USER. The way you would describe the config there would be the following:

import { Config } from "effect"

const secrets = Config.nested("secrets")
const mysqlUser = secrets(Config.secret("MYSQL_USER"))

mysqlUser becomes a nested config resulting in the /secrets/MYSQL_USER file being loaded by the config provider (the leading / in the path is the default rootPath config, remember?)

My current worry is that the above config provider is a platform-specific implementation. The Effect core library only seems to contain platform-agnostic code. So a better fit in terms of its distribution might be the @effect/platform package? That doesn't sound ideal either because all the other config providers are part of the core library. What would you suggest, how should I approach this problem?

What alternatives have you considered?

None.

[tim-smart]

We could look at moving the cli ConfigFile module to platform: https://github.com/Effect-TS/effect/blob/main/packages/cli/test/ConfigFile.test.ts#L13

[vecerek]

We could look at moving the cli ConfigFile module to platform: https://github.com/Effect-TS/effect/blob/main/packages/cli/test/ConfigFile.test.ts#L13

ConfigFile, in its current implementation, seems quite limiting (json, ini, yaml, toml). Often, secrets are stored in individual files, containing the value only. Should ConfigFile be lifted to platform, at least a raw option would also have to be added. Personally, I would prefer if parsing was left out from a config file provider (at least initially). I see parsing only as an extension to providing the file contents. It's something that could be solved through composition. Nonetheless, a higher-level abstraction could then be created but I'd like to see a lower level API first. That could also give you a chance to discover common use cases, which could then feed the decision-making process around the API of that higher level abstraction.

[tim-smart]

Looks like they solve different problems. Config files vs Config file trees. We could add support for both.