Minimum Viable Schema Protocol

[jamiebuilds]

TL;DR

I want to propose a "minimum viable protocol" for libraries to accept schemas as part of their API without depending directly on Valibot (or another library such as Zod)

// Protocol:
export let parseSymbol: unique symbol = Symbol.for("https://github.com/fabian-hiller/symbol-parse")
/** @throws {unknown} If the input is rejected. */
export type Parse<T> = (input: unknown) => T
export type Schema<T> = { [parseSymbol]: Parse<T> }

// Example library:
export async function fetchJson<T>(endpoint: string, schema: Schema<T>): Promise<T> {
  let response = await fetch(endpoint)
  let json = await response.json()
  try {
    // Call `parse()` on some unknown data and get the schema's output type
    return schema[parseSymbol](json)
  } catch (error: unknown) {
    // Catch whatever error was thrown by the schema
    throw new Error("Unexpected data", { cause: error })
  }
}

Background

I'm building a library that accept a Valibot schema's as part of its API.

import { sql } from "lib"
import * as v from "valibot"

function getUserAge(userId: string): number {
  return sql`select age from user where id = ${userId}`
    .pluck(v.number())
    .one()
}

For the purposes of this library, I don't really need to do much of anything with Valibot itself. I'm not creating my own schemas and I just want to give users nice typings and convenient APIs when using their own schemas.

To make this work, I just need to be able to call Valibot's parse(schema, input) method.

Problem

It would be nice if I didn't have to pull in Valibot as a dependency just for its most basic functionality. It would then mean I also have to keep it up to date for major versions.

It would also be nice if I could just say that I accept several different schema libraries.

Solution

If Valibot could commit to supporting (across major versions of Valibot) a well-specified, minimum viable protocol similar to that of Promises .then, Iterators [Symbol.iterator], or Observables [Symbol.observable], then library authors could depend just on the protocol and not need to pull in Valibot as a dependency.

It would also make these libraries generic across any other schema library that wanted to implement the protocol.

There could be a package similar to https://github.com/benlesh/symbol-observable that just exposes a shared symbol:

export let parseSymbol: unique symbol = Symbol.for("https://github.com/fabian-hiller/symbol-parse")
/** @throws {unknown} If the input is rejected. */
export type Parse<T> = (input: unknown) => T
export type Schema<T> = { [parseSymbol]: Parse<T> }

Then in libraries they can just make use of it in their types:

import { parseSymbol, Schema } from "symbol-parse"

function parse<T>(schema: Schema<T>, input: unknown): T {
  return schema[parseSymbol](input)
}

Why "minimum viable" protocol?

Asking lots of developers to coordinate around a shared protocol can be like herding cats. A larger footprint becomes harder to come to consensus on and asks more of everyone who interacts with it.

For this purpose I suggest:

• No separate Input or Output types
  • I don't think this would be particularly useful for libraries that want to be generic over multiple schema libraries/versions.
• No Issues or other error/rejection types
  • If libraries really want to introspect the issues types, they should just depend on a schema library directly
• No safeParse() or SafeParseResult
  • If Issues isn't typed, there really isn't much of a purpose for this at all. Library authors can just use try-catch (if they even need to) and that helps reduces the scope of this protocol.

[fabian-hiller]

In general, I welcome such ideas. However, the main problem for Valibot will be that following a protocol will unnecessarily increase the bundle size for all users not using that protocol.

As a workaround for now. Valibot provides a parser and safeParser method that returns a function. If your users wrap their schemas in parser or safeParser, you can run them directly without adding Valibot as a dependency. Another workaround that many form libraries use is adapters and resolvers.

import * as v from 'valibot';

const parseNumber = v.parser(v.number());

const number1 = parseNumber(123); // returns 123
const number2 = parseNumber('123'); // throws error

I suggest you also have a look at TypeSchema, Standard Schema and this discussion.

[jamiebuilds]

I would be surprised if this had much of a real world impact, it's not like Valibot gets used without parse()/safeParse() (which are called by parser()/safeParser()) anyways which already includes the code for global config and such. If you abstracted it away to a function call in all of the schema factory functions, it's not much more code than the parse functions that every user has to include anyways.

export function string() {
  return defineSchema({ ... })
}
// or
export let string = defineSchema(() => {
  // ...
})

You honestly might want to do something like that with these factories anyways because you could introduce some caching which could improve the performance of Valibot a lot.

[fabian-hiller]

It makes a difference because parse needs to import ValiError and if people only use safeParse this code will never be used. It is true that the real world impact may be small, but it still feels wrong to me because it goes against the philosophy of our API design and implementation. If all the other libraries follow such a specification, Valibot will probably follow too, but Valibot is the wrong library to start such an initiative.

You honestly might want to do something like that with these factories anyways because you could introduce some caching which could improve the performance of Valibot a lot.

Create idea! I will investigate this as part of #572.

[jamiebuilds]

If such a proposal instead was an equivalent of safeParse() and specified a return value of:

type Issue = { path?: PropertyKey[], message: string }
type Result<T> = 
  | { ok: true, result: T, issues: void }
  | { ok: false, result: void, issues: Issue[] }

export let parseSymbol: unique symbol = Symbol.for("parse")
/** @throws {unknown} If the input is rejected. */
export type Parse<T> = (input: unknown) => Result<T>
export type Schema<T> = { [parseSymbol]: Parse<T> }

Would that be more acceptable?

[fabian-hiller]

Yes, but supporting this format will probably add additional code besides safeParse, which will increase the size of the bundle even more. I support your idea, and we should discuss it as a community, but Valibot will probably adopt it later than other libraries due to our focus on bundle size and modularity.

[jamiebuilds]

Yeah, I'm discussing it with you now to understand what version of this protocol you'd accept. It's possible that it could include things that reduce your implementation even further:

• Allow additional properties on Result and Issue
  • Simplifies the implementation so you don't need to map your names of properties to the spec's, you could just use the same names and return the full object you need.
• Don't require an ok/success/valid parameter on Result ({ value: T, issues?: Issue })
  • You do this in Valibot by mapping success: !dataset.issues
  • Consumers of the protocol could do the same

Together this would mean the implementation of this spec is no more than:

let decorate = schema => {
  schema[parseSymbol] = input => {
    return schema._run(input, getGlobalConfig())
  }
}

// or minified (maybe code-golf-able further in context)

let d=s=>s[p]=i=>s._run(i,c())

Or going even further, if you wanted to be able to just replace _run with a symbol (or whatever the spec wants to use for its property:

• Allow additional parameters to the parse() method
  • This would allow you to accept config as an optional second argument and even call getGlobalConfig()

That would make this the full change in Valibot:

  export function number(
    message?: ErrorMessage<NumberIssue>
  ): NumberSchema<ErrorMessage<NumberIssue> | undefined> {
    return {
      kind: 'schema',
      type: 'number',
      reference: number,
      expects: 'number',
      async: false,
      message,
-     _run(dataset, config) {
+     [parseSymbol](dataset, config = getConfig()) {
        if (typeof dataset.value === 'number' && !isNaN(dataset.value)) {
          dataset.typed = true;
        } else {
          _addIssue(this, 'type', dataset, config);
        }
        return dataset as Dataset<number, NumberIssue>;
      },
    };
  }

Which minified is a negligible amount of bytes other than getConfig() which always has to be used in Valibot since its included in all of the parse methods.

[jamiebuilds]

Wrote a proposal over here: https://github.com/standard-schema/standard-schema/issues/3

[fabian-hiller]

Thank you! The [parseSymbol](dataset, config = getConfig()) ... change looks much better for Valibot. One thing to note is that dataset contains more info then the raw input. So I am not sure if this will work the same way with other schema libraries.

Wrote a proposal over here: https://github.com/standard-schema/standard-schema/issues/3

Thanks! I will have a look at it.

[fabian-hiller]

I think I can close this issue. Further discussion should take place in the Standard Schema repo: https://github.com/standard-schema/standard-schema