Add `object` support to Config for Cloudflare Integration and Declarative Services

[mw10013]

What is the problem this feature would solve?

Traditionally, environment variables are string based. Modern runtimes such as Cloudflare's also put objects into the environment. Cloudflare calls these bindings. They allow a Worker, Cloudflare's serverless function, to interact with resources on Cloudflare such as D1 (sqlite), R2 (object storage), and DO (durable objects). ConfigProviders hew to tradition and support only string-based configuration so they cannot fully support Cloudflare envs.

In Effect, a service can be 'declarative' — meaning it avoids runtime boilerplate — if its layer constructors don’t need arguments from the application. This is a desirable property, as it simplifies layer construction. Config supports this declarativity when limited to string-based configuration. However, a Cloudflare service, which depends on object bindings in the environment, can’t leverage Config to achieve this property.

I propose adding object support to Config for full Cloudflare env integration and to unlock declarative Cloudflare services.

What is the feature you are proposing to solve the problem?

Proposed API

ConfigProvider.fromObject: <T extends { [K in keyof T]: string | object }>(object: T) => Layer<never, never, never>

Config.object: (name: string) => Config<object>

Proof of Concept (POC)

The POC uses pathological type assertions to mock the proposed API, demonstrating that the existing machinery in Effect can support this. It is in this repo and its essence:

// ConfigEx.ts

// Mock of ConfigProvider.fromObject: <T extends { [K in keyof T]: string | object }>(object: T) => Layer<never, never, never>
export const fromObject = <T extends { [K in keyof T]: string | object }>(object: T) =>
	pipe(
		object as unknown as Record<string, string>,
		Record.toEntries,
		(tuples) => new Map(tuples),
		ConfigProvider.fromMap,
		Layer.setConfigProvider
	)

// Mock of Config.object: (name: string) => Config<object>
export const object = (name: string) =>
	Config.string(name).pipe(
		Config.mapOrFail((value) =>
			value !== null && typeof value === 'object'
				? Either.right(value as object)
				: Either.left(ConfigError.InvalidData([], `Expected an object but received ${value}`))
		)
	)

Declarative Cloudflare service with a dependency on another declarative Cloudflare service:

// Poll.ts
export class Poll extends Effect.Service<Poll>()('Poll', {
	accessors: true,
	dependencies: [KV.Default],
	effect: Effect.gen(function* () {
		const pollDo = yield* ConfigEx.object('POLL_DO').pipe(
			Config.mapOrFail((object) =>
				Predicate.hasProperty(object, 'idFromName') && typeof object.idFromName === 'function'
					? Either.right(object as Env['POLL_DO'])
					: Either.left(ConfigError.InvalidData([], `Expected a DurableObjectNamespace but received ${object}`))
			)
		)
		// Snip
		return {
			// Snip
		}
	})
}) {}

The runtime

// index.tsx
export const makeRuntime = (env: Env) => {
	const ConfigLive = ConfigEx.fromObject(env)
	return Layer.mergeAll(Poll.Default).pipe(Layer.provide(ConfigLive), ManagedRuntime.make)
}

What alternatives have you considered?

No response

[johtso]

I was wondering how to deal with this too! I was thinking I might just have to provide the simple string based values as Config, but pass the bindings explicitly to their relevant service layers when constructing the runtime.

Out of curiosity, do you build the runtime from scratch for every request? I was thinking you could reuse it in a "hot" worker, but the fact that the config and bindings in env can change request to request made me think twice..

[johtso]

@mw10013 https://discord.com/channels/795981131316985866/1381408104880668712/1381572651671228426

Would be interested to hear your thoughts if you have time

[mw10013]

@johtso I'm still a newbie at effect and cloudflare and you seem further along. I just used the pathological type assertion to get going with a proof-of-concept (POC) integrating effect with cloudflare, hono, openauth, and react router in one worker function to see how it feels. It feels terrific since declarative effect services are joy unbounded.

When this breaks in the future, I'll be forced to remove the object bindings from Config. That will be a sad day since Config is world-class, overridable, and ubiquitous (indispensable for declarative services).

Re: "hot" or not. I'm surprised to hear that the env changes from request to request. My understanding is that changes to the env required a deployment for the changes to take effect. In the POC, I create the ManagedRuntime in the request.

Would appreciate if you shared the solution you eventually land upon so I can learn. There seems little interest or appetite to enhance Config to accommodate objects so I will need to follow in your footsteps.