Schema: expose `literals` and `fields`

[patroza]

What is the problem this feature would solve?

Exposing literals means you can define a literal schema and then use the (fully typed and ordered) members conveniently from anywhere else as values.

export const SupportedCountries = literal("DE", "CH", "AT")

console.log(`You may provide any of the following countries: ${SupportedCountries.literals.join(", ")}`)

Exposing fields means you can easily merge, pick and omit from them to create new schemas. Especially useful if you simply want the fields, and not extend existing classes, or creating A & B & C chains. The benefit over pick, omit, extend is that you can use standard object merging (spread/destructure) to build new structs/classes.

const struct1 = struct({ a: number, b: optional(string), c: string })
const struct2 = struct({ d: boolean, e: optional(number), f: literal("g", "h") })
const struct3 = struct({ ...struct1.fields, ...pipe(struct2.fields, omit("e")) })
const struct4 = struct({ ...struct1.fields, ...pipe(struct2.fields, pick("d", "f")) })

What is the feature you are proposing to solve the problem?

See for an example https://github.com/Effect-TS/effect/pull/2013: https://github.com/Effect-TS/effect/pull/2013/files#diff-7da43f2b7b3dea1554f2627482ba41e4f885c908848b72704d9887ffcc065153 I know there's the limitation we don't carry the fields and literals onward when we use combinators. so far this limitation hasn't impeded it's use for me. However perhaps there are alternatives to this, which reach into the AST and retrieve the values for us instead?

Let me know what you think, and we can update the expected types tests etc accordingly.

What alternatives have you considered?

No response

[gcanti]

There is the risk that the .d.ts files explode, I had some issues in the early versions of io-ts

import * as S from "@effect/schema/Schema"
import type { Simplify } from "effect/Types"

declare const struct: <Fields extends S.StructFields>(
  fields: Fields
) =>
  & S.Schema<S.Schema.Context<Fields[keyof Fields]>, Simplify<S.FromStruct<Fields>>, Simplify<S.ToStruct<Fields>>>
  & { readonly fields: Fields }

const schema1 = struct({ a: S.string, b: S.number })
const schema2 = struct({ c: schema1 })
const schema3 = struct({ d: schema2, e: schema1 })
/*
const schema3: S.Schema<never, {
    readonly d: {
        readonly c: {
            readonly a: string;
            readonly b: number;
        };
    };
    readonly e: {
        readonly a: string;
        readonly b: number;
    };
}, {
    readonly d: {
        readonly c: {
            readonly a: string;
            readonly b: number;
        };
    };
    readonly e: {
        readonly a: string;
        readonly b: number;
    };
}> & {
    readonly fields: {
        d: S.Schema<never, {
            readonly c: {
                readonly a: string;
                readonly b: number;
            };
        }, {
            readonly c: {
                readonly a: string;
                readonly b: number;
            };
        }> & {
            readonly fields: {
                c: S.Schema<never, {
                    readonly a: string;
                    readonly b: number;
                }, {
                    readonly a: string;
                    readonly b: number;
                }> & {
                    readonly fields: {
                        a: S.Schema<never, string, string>;
                        b: S.Schema<never, number, number>;
                    };
                };
            };
        };
        e: S.Schema<never, {
            readonly a: string;
            readonly b: number;
        }, {
            readonly a: string;
            readonly b: number;
        }> & {
            readonly fields: {
                a: S.Schema<never, string, string>;
                b: S.Schema<never, number, number>;
            };
        };
    };
}
*/

[patroza]

There is the risk that the .d.ts files explode, I had some issues in the early versions of io-ts

that's however a risk with any kind of non-opaque struct schema already, and a reason why it's important for people to embrace opaque types.

that said, I would also be fine with exposing a combinator who extract fields instead, but we will loose PropertyDescriptors etc.

[gcanti]

Well, the amount of information that a schema carries around definitely matters. The issue here is that with this modification, the size of the .d.ts file grows quadratically as the nesting levels increase

import * as S from "@effect/schema/Schema"
import type { Simplify } from "effect/Types"

declare const struct: <Fields extends S.StructFields>(
  fields: Fields
) => S.Schema<S.Schema.Context<Fields[keyof Fields]>, Simplify<S.FromStruct<Fields>>, Simplify<S.ToStruct<Fields>>> & {
  readonly fields: Fields
}

const schema1 = struct({ a: S.string })
const schema2 = struct({ b: schema1 })
const schema3 = struct({ c: schema2 })
const schema4 = struct({ d: schema3 })
const schema5 = struct({ e: schema4 })
export const schema6 = struct({ e: schema5 }) // 6 levels

.d.ts

import * as S from "@effect/schema/Schema";
export declare const schema6: S.Schema<never, {
    readonly e: {
        readonly e: {
            readonly d: {
                readonly c: {
                    readonly b: {
                        readonly a: string;
                    };
                };
            };
        };
    };
}, {
    readonly e: {
        readonly e: {
            readonly d: {
                readonly c: {
                    readonly b: {
                        readonly a: string;
                    };
                };
            };
        };
    };
}> & {
    readonly fields: {
        e: S.Schema<never, {
            readonly e: {
                readonly d: {
                    readonly c: {
                        readonly b: {
                            readonly a: string;
                        };
                    };
                };
            };
        }, {
            readonly e: {
                readonly d: {
                    readonly c: {
                        readonly b: {
                            readonly a: string;
                        };
                    };
                };
            };
        }> & {
            readonly fields: {
                e: S.Schema<never, {
                    readonly d: {
                        readonly c: {
                            readonly b: {
                                readonly a: string;
                            };
                        };
                    };
                }, {
                    readonly d: {
                        readonly c: {
                            readonly b: {
                                readonly a: string;
                            };
                        };
                    };
                }> & {
                    readonly fields: {
                        d: S.Schema<never, {
                            readonly c: {
                                readonly b: {
                                    readonly a: string;
                                };
                            };
                        }, {
                            readonly c: {
                                readonly b: {
                                    readonly a: string;
                                };
                            };
                        }> & {
                            readonly fields: {
                                c: S.Schema<never, {
                                    readonly b: {
                                        readonly a: string;
                                    };
                                }, {
                                    readonly b: {
                                        readonly a: string;
                                    };
                                }> & {
                                    readonly fields: {
                                        b: S.Schema<never, {
                                            readonly a: string;
                                        }, {
                                            readonly a: string;
                                        }> & {
                                            readonly fields: {
                                                a: S.Schema<never, string, string>;
                                            };
                                        };
                                    };
                                };
                            };
                        };
                    };
                };
            };
        };
    };
};

current .d.ts (growth is linear)

import * as S from "@effect/schema/Schema";
export declare const schema6: S.Schema<never, {
    readonly e: {
        readonly e: {
            readonly d: {
                readonly c: {
                    readonly b: {
                        readonly a: string;
                    };
                };
            };
        };
    };
}, {
    readonly e: {
        readonly e: {
            readonly d: {
                readonly c: {
                    readonly b: {
                        readonly a: string;
                    };
                };
            };
        };
    };
}>;

[gcanti]

it's important for people to embrace opaque types

I'm not sure what you mean by "opaque types" (whether you're referring to this trick: link or using S.Class?), but the amount of information that goes into the .d.ts file remains the same.

I would also be fine with exposing a combinator who extract fields instead

Mmh, I'm not sure I'm following, how can we extract the typed fields if we don't have their types stored somewhere at the type level (as you did with & { fields: Fields })?

[patroza]

I'm not sure what you mean by "opaque types" (whether you're referring to this trick: link or using S.Class?), but the amount of information that goes into the .d.ts file remains the same.

yes class https://github.com/effect-ts-app/boilerplate/blob/main/_project/models/_src/User.ts#L71 (ExtendedClass), or struct retyped with interfaces inside From and To. regarding the data that goes into the .d.ts; no it will stick to one Level. all these nested From and To, just turn into: OpaqueFrom and OpaqueTo for each field.

e.g:

type User =
& Schema<never, { readonly name: FullName.From; }, { readonly name: FullName }>
& { fields: { readonly name: typeof FullName } }

(for struct indeed it will still carry the fields nested, but also will still benefit when the members are opaque)

I would also be fine with exposing a combinator who extract fields instead

Mmh, I'm not sure I'm following, how can we extract the typed fields if we don't have their types stored somewhere at the type level (as you did with & { fields: Fields })?

yea this is what I mentioned, for each field we can get the Schema<ROfParent, From, To> out, but not Schema<ROfField, From, To>, nor PropertyDescriptor<....> mmh but I guess renames are a problem :)

Well, the amount of information that a schema carries around definitely matters

no disagreement here.

[gcanti]

@patroza A technique I've successfully used in io-ts to avoid excessive nesting is to define an interface for each main API to save the type of the arguments:

import type * as AST from "@effect/schema/AST"
import * as S from "@effect/schema/Schema"
import type { Simplify } from "effect/Types"

type AnySchema<R = unknown> = S.Schema<any, any, R> | S.Schema<never, never, R>

interface struct<Fields extends S.StructFields>
  extends S.Schema<Simplify<S.ToStruct<Fields>>, Simplify<S.FromStruct<Fields>>, S.Schema.Context<Fields[keyof Fields]>>
{
  readonly fields: Fields
}

declare const struct: <Fields extends S.StructFields>(fields: Fields) => struct<Fields>

interface array<Item extends AnySchema>
  extends S.Schema<ReadonlyArray<S.Schema.To<Item>>, ReadonlyArray<S.Schema.From<Item>>, S.Schema.Context<Item>>
{
  readonly item: Item
}

declare const array: <Item extends AnySchema>(item: Item) => array<Item>

interface union<Members extends ReadonlyArray<AnySchema>>
  extends S.Schema<S.Schema.To<Members[number]>, S.Schema.From<Members[number]>, S.Schema.Context<Members[number]>>
{
  readonly members: Members
}

declare const union: <Members extends ReadonlyArray<AnySchema>>(...members: Members) => union<Members>

interface literal<Literals extends ReadonlyArray<AST.LiteralValue>>
  extends union<{ readonly [I in keyof Literals]: S.Schema<Literals[I]> }>
{
  readonly literals: Literals
}

declare const literal: <Literals extends ReadonlyArray<AST.LiteralValue>>(...literals: Literals) => literal<Literals>

/*
const s1: struct<{
    a: S.Schema<string, string, never>;
    b: array<S.Schema<number, string, never>>;
    c: literal<["a", "b"]>;
}>
*/
export const s1 = struct({
  a: S.string,
  b: array(S.NumberFromString),
  c: literal("a", "b")
})

// const numberFromString: S.Schema<number, string, never>
export const numberFromString = s1.fields.b.item

// const myliterals: ["a", "b"]
export const myliterals = s1.fields.c.literals

IMO it's also more readable because there are no repetitions:

declare const asSchema: <S extends AnySchema>(
  schema: S
) => S.Schema<S.Schema.To<S>, S.Schema.From<S>, S.Schema.Context<S>>

/*
const s1AsSchema: S.Schema<{
    readonly a: string;
    readonly b: readonly number[];
    readonly c: "a" | "b";
}, {
    readonly a: string;
    readonly b: readonly string[];
    readonly c: "a" | "b";
}, never>
*/
export const s1AsSchema = asSchema(s1)

i.e.

const s1: struct<{
    a: S.Schema<string, string, never>;
    b: array<S.Schema<number, string, never>>;
    c: literal<["a", "b"]>;
}>

versus

const s1AsSchema: S.Schema<{
    readonly a: string;
    readonly b: readonly number[];
    readonly c: "a" | "b";
}, {
    readonly a: string;
    readonly b: readonly string[];
    readonly c: "a" | "b";
}, never>

If desired, and solely for the purpose of readability, we could go even further and define helper interfaces to hide unnecessary type parameters (i.e., when R = never and/or I = A)

interface S<A> extends S.Schema<A> {}
interface SI<A, I> extends S.Schema<A, I> {}

type SymplifySchema<Schema extends AnySchema> = S.Schema.Context<Schema> extends never
  ? Equals<S.Schema.From<Schema>, S.Schema.To<Schema>> extends true ? S<S.Schema.To<Schema>>
  : SI<S.Schema.To<Schema>, S.Schema.From<Schema>>
  : Schema

declare const string: SymplifySchema<typeof S.string>
declare const NumberFromString: SymplifySchema<typeof S.NumberFromString>

/*
const s1Simplified: struct<{
    a: S<string>;
    b: array<SI<number, string>>;
    c: literal<["a", "b"]>;
}>
*/
export const s1Simplified = struct({
  a: string,
  b: array(NumberFromString),
  c: literal("a", "b")
})

i.e.

const s1Simplified: struct<{
    a: S<string>;
    b: array<SI<number, string>>;
    c: literal<["a", "b"]>;
}>

versus

const s1: struct<{
    a: S.Schema<string, string, never>;
    b: array<S.Schema<number, string, never>>;
    c: literal<["a", "b"]>;
}>

---

example with nested fields

const schema1 = struct({ a: string })
const schema2 = struct({ b: schema1 })
const schema3 = struct({ c: schema2 })
const schema4 = struct({ d: schema3 })
const schema5 = struct({ e: schema4 })

/*
const schema6: struct<{
  e: struct<{
      e: struct<{
          d: struct<{
              c: struct<{
                  b: struct<{
                      a: S<string>;
                  }>;
              }>;
          }>;
      }>;
  }>;
*/
export const schema6 = struct({ f: schema5 }) // 6 levels

// const leaf: S<string>
export const leaf = schema6.fields.f.fields.e.fields.d.fields.c.fields.b.fields.a

[gcanti]

For the sake of readability, we could define a lot of interfaces (as I did in io-ts), essentially one for each built-in schema / combinator (even for S.string or S.NumberFromString for example):

interface $string extends S.Schema<string> {}
declare const string: $string

interface NumberFromString extends S.Schema<number, string> {}
declare const NumberFromString: NumberFromString

interface option<V extends AnySchema>
  extends S.Schema<O.Option<S.Schema.To<V>>, O.Option<S.Schema.From<V>>, O.Option<S.Schema.Context<V>>>
{
  readonly value: V
}
declare const option: <V extends AnySchema>(v: V) => option<V>

/*
const schema: struct<{
    a: $string;
    b: array<NumberFromString>;
    c: literal<["a", "b"]>;
    d: option<$string>;
}>
*/
export const schema = struct({
  a: string,
  b: array(NumberFromString),
  c: literal("a", "b"),
  d: option(string)
})

// const optionValue: $string
const optionValue = schema.fields.d.value

[patroza]

@gcanti yes I like it, or in @mikearnaldi speak “what a great idea!” ;) It’s simply the wholesale continuation of Opaqueness, including the Schema. Typewise i suspect its much better performance wise but can’t say without benchmarking first.

An example with property descriptors would be nice, and eg prop mapping (different name in From) if we would implement like https://github.com/patroza/effect/pull/3/files#diff-7da43f2b7b3dea1554f2627482ba41e4f885c908848b72704d9887ffcc065153R4830. (if im not mistaken, the benefit also is that picking these out of fields compared to current S.pick gets you the mapping transformation out too.)

I wonder if going all in or still eg exposing R would be useful somehow to make them stand out.

[gcanti]

if im not mistaken, the benefit also is that picking these out of fields compared to current S.pick gets you the mapping transformation out too

@patroza yes, this is an example using my working branch (https://github.com/Effect-TS/effect/pull/2172)

import * as S from "@effect/schema/Schema"

/*
const schema: S.struct<{
    a: S.Schema<string, string, never>;
    b: S.PropertySignature<"c", ":", number, ":", string, never>;
}>

i.e.

S.Schema<{
    readonly a: string;
    readonly c: number;
}, {
    readonly a: string;
    readonly b: string;
}, never>

*/
const schema = S.struct({
  a: S.string,
  b: S.propertySignatureDeclaration(S.NumberFromString).pipe(S.propertySignatureKey("c"))
})

console.log(S.decodeSync(schema)({ a: "a", b: "1" })) // => { a: 'a', c: 1 }

/*
const schema2: S.struct<{
    d: S.Schema<boolean, boolean, never>;
    a: S.Schema<string, string, never>;
    b: S.PropertySignature<"c", ":", number, ":", string, never>;
}>

i.e.

S.Schema<{
    readonly a: string;
    readonly c: number;
    readonly d: boolean;
}, {
    readonly a: string;
    readonly b: string;
    readonly d: boolean;
}, never>

*/
const schema2 = S.struct({
  ...schema.fields,
  d: S.boolean
})

console.log(S.decodeSync(schema2)({ a: "a", b: "1", d: true })) // => { d: true, a: 'a', c: 1 }

[gcanti]

For the sake of readability...

Update from the working branch

import * as S from "@effect/schema/Schema"

/*
const current: S.struct<{
    a: S.literal<["A"]>;
    b: S.literal<["B", "C"]>;
    c: S.NumberFromString;
    d: S.PropertySignature<never, "?:", number | undefined, "?:", number | undefined, never>;
    e: S.tuple<[S.$string, S.$number]>;
    f: S.array<S.$string>;
    g: S.union<[S.$string, S.$number]>;
    h: S.nonEmptyArray<S.$string>;
    i: S.struct<{
        a: S.$string;
    }>;
}>
*/
export const current = S.struct({
  a: S.literal("A"),
  b: S.literal("B", "C"),
  c: S.NumberFromString,
  d: S.optional(S.number),
  e: S.tuple(S.string, S.number),
  f: S.array(S.string),
  g: S.union(S.string, S.number),
  h: S.nonEmptyArray(S.string),
  i: S.struct({
    a: S.string
  })
})

/*
const old: S.Schema<{
    readonly a: "A";
    readonly b: "B" | "C";
    readonly c: number;
    readonly e: readonly [string, number];
    readonly f: readonly string[];
    readonly g: string | number;
    readonly h: readonly [string, ...string[]];
    readonly i: {
        readonly a: string;
    };
    readonly d?: number | undefined;
}, {
    readonly a: "A";
    readonly b: "B" | "C";
    readonly c: string;
    readonly e: readonly [string, number];
    readonly f: readonly string[];
    readonly g: string | number;
    readonly h: readonly [string, ...string[]];
    readonly i: {
        readonly a: string;
    };
    readonly d?: number | undefined;
}, never>
*/
export const old = S.asSchema(current)

[patroza]

@gcanti wow, that looks great!

b: S.propertySignatureDeclaration(S.NumberFromString).pipe(S.propertySignatureKey("c"))

one thing I would ponder on is, I personally like to think of my To schema as the center of the universe, not my From. I mean this seems a little counter intuitive when one thinks logically about From->To, but still. In my current experimentation, I use a MapFrom, not MapTo. so I would define C, maps from B, not B maps to C. it's in the same vein as "Number" From "String" though. "C" from "B"

You don't say "Number" To "String", so neither "B" to "C" imo

[gcanti]

The name aligns with the type: NumberFromString: Schema<number, string> (number from string), but for the transformations, for now, I've done it this way because it seems to me that people prefer to think of transformations as From -> To, all the other transformation APIs are designed this way (e.g. transform(from, to, ...), transformOrFail(from, to, ...), transformLiteral(from, to))

[patroza]

The name aligns with the type: NumberFromString: Schema<number, string> (number from string), but for the transformations, for now, I've done it this way because it seems to me that people prefer to think of transformations as From -> To, all the other transformation APIs are designed this way (e.g. transform(from, to, ...), transformOrFail(from, to, ...), transformLiteral(from, to))

I think for the apis it might make sense, but I strongly believe when you define a struct or a class, the properties define the To (aka the type/schema you're actually defining).

I find it very strange to say person.a, while person has a b field somehow mapped to a

[gcanti]

@patroza yeah, you are right, moving the key to 'from' makes it much clearer

b: S.PropertySignature<":", number, "c", ":", string, never>;

it's like

b: number <- c: string

Updated example

import * as S from "@effect/schema/Schema"

/*
const schema: S.struct<{
    a: S.Schema<string, string, never>;
    b: S.PropertySignature<":", number, "c", ":", string, never>;
}>

i.e.

S.Schema<{
    readonly a: string;
    readonly b: number;
}, {
    readonly a: string;
    readonly c: string;
}, never>

*/
const schema = S.struct({
  a: S.string,
  b: S.propertySignatureDeclaration(S.NumberFromString).pipe(S.propertySignatureKey("c"))
})

console.log(S.decodeSync(schema)({ a: "a", c: "1" })) // => { a: 'a', b: 1 }

/*
const schema2: S.struct<{
    d: S.Schema<boolean, boolean, never>;
    a: S.Schema<string, string, never>;
    b: S.PropertySignature<":", number, "c", ":", string, never>;
}>

i.e.

S.Schema<{
    readonly a: string;
    readonly b: number;
    readonly d: boolean;
}, {
    readonly a: string;
    readonly c: string;
    readonly d: boolean;
}, never>

*/
const schema2 = S.struct({
  ...schema.fields,
  d: S.boolean
})

console.log(S.decodeSync(schema2)({ a: "a", c: "1", d: true })) // => { d: true, a: 'a', b: 1 }

[gcanti]

Resolved in @effect/schema 0.64.x