High-level explanation of API changes

[mmkal]

📖 Documentation

Would it be possible to add a paragraph or two to the readme, on the differences between "experimental" (Decoder/Encoder/Codec/Schema) and "stable" (Type), and who should use which? Assuming the concept of Type will be removed in v3, does it mean usage like

import * as t from 'io-ts'

const Person = t.type({ name: t.string })

Person.decode({ name: 'Alice' })

Will need to change? If so, will there be a migration guide (I assume Codec largely replaces Type)? I've seen the S.make(S => ...) syntax around in a few places, but it's not immediately clear if applications relying on io-ts will need to use it, or if it's mainly a low-level construct.

A link to a tracking issue could also work.

[gcanti]

Decoder, Encoder and Codec modules are published as experimental in order to get early feedback from the community.

Codec should largely replace Type except for untagged unions (that's because encoders don't support them).

The Schema module is an advanced feature which must prove its worth, the goal would be to express a generic schema and then derive from that multiple concrete instances (like decoders, encoders, equality instances, arbitraries, etc...).

I think that speaking about a v3 is too early, the new modules / APIs must be validated first.

I'm reserving the 2.2 label to track the issues related to the new modules.

For what concerns the APIs changes, here's a tiny migration guide in order to help experimenting with the Decoder module:

• Decoder is defined with only one type parameter
• keyof is replaced by literal
• record doesn't accept a domain schema anymore
• brand is replaced by refinement / parse which are not opinionated on how you want to define your branded types
• recursive renamed to lazy (mutually recursive decoders are supported)
• intersect is now pipeeable
• tagged unions must be explicitly defined (with sum) in order to get optimized
• tuple is not limited to max five components

[mmkal]

Thanks - some follow-up questions:

Codec should largely replace Type except for untagged unions (that's because encoders don't support them).

Will untagged unions be supported at all?

• Decoder is defined with only one type parameter

Does that mean there's no longer such a thing as a Decoder from I to A - effectively I is always unknown?

• keyof is replaced by literal

Will this affect the advice to use t.keyof({ a: null, b: null }) to achieve enum-like behaviour (potentially related to the answer to "Will untagged unions be supported at all")?

• brand is replaced by refinement / parse which are not opinionated on how you want to define your branded types

Does that mean anything for https://github.com/gcanti/io-ts/issues/373? The request there was to keep un-branded refinements as an option.

[steida]

Will this be possible as Codec? I mean, the possibility to decode from two different types.

export const FaunaDocRef = (() => {
  const Serialized = t.type({
    '@ref': t.type({
      id: FaunaID,
      collection: t.type({
        '@ref': t.type({
          id: t.string,
          collection: t.type({
            '@ref': t.type({ id: t.literal('collections') }),
          }),
        }),
      }),
    }),
  });
  const FaunaDocRef = t.type({
    id: FaunaID,
    collection: t.string,
  });
  type FaunaDocRef = t.TypeOf<typeof FaunaDocRef>;
  return new t.Type<FaunaDocRef, values.Ref, unknown>(
    'FaunaDocRef',
    FaunaDocRef.is,
    (u, c) => {
      if (u instanceof values.Ref) {
        return u.collection
          ? t.success({
              id: u.id as FaunaID, // as FaunaID is ok, we don't create ids anyway
              collection: u.collection.id,
            })
          : t.failure(u, c);
      }
      return either.either.chain(Serialized.validate(u, c), (s) =>
        t.success({
          id: s['@ref'].id,
          collection: s['@ref'].collection['@ref'].id,
        }),
      );
    },
    (a) =>
      new values.Ref(
        a.id,
        new values.Ref(a.collection, values.Native.COLLECTIONS),
      ),
  );
})();
export type FaunaDocRef = t.TypeOf<typeof FaunaDocRef>;

[gcanti]

Will untagged unions be supported at all?

They are supported in Decoder. I can't find a reliable way to support them in Encoder though.

So either you define an encoder by hand...

import { left, right } from 'fp-ts/lib/Either'
import * as C from 'io-ts/lib/Codec'
import * as D from 'io-ts/lib/Decoder'
import * as G from 'io-ts/lib/Guard'

const NumberFromString: C.Codec<number> = C.make(
  D.parse(D.string, (s) => {
    const n = parseFloat(s)
    return isNaN(n) ? left(`cannot decode ${JSON.stringify(s)}, should be NumberFromString`) : right(n)
  }),
  { encode: String }
)

export const MyUnion: C.Codec<number | string> = C.make(D.union(NumberFromString, D.string), {
  encode: (a) => (G.string.is(a) ? a : NumberFromString.encode(a))
})

...or you extend Codec to something containing a guard

import * as E from 'io-ts/lib/Encoder'

interface Compat<A> extends D.Decoder<A>, E.Encoder<A>, G.Guard<A> {}

function make<A>(codec: C.Codec<A>, guard: G.Guard<A>): Compat<A> {
  return {
    is: guard.is,
    decode: codec.decode,
    encode: codec.encode
  }
}

function union<A extends ReadonlyArray<unknown>>(
  ...members: { [K in keyof A]: Compat<A[K]> }
): Compat<A[number]> {
  return {
    is: G.guard.union(...members).is,
    decode: D.decoder.union(...members).decode,
    encode: (a) => {
      for (const member of members) {
        if (member.is(a)) {
          return member.encode(a)
        }
      }
    }
  }
}

const string = make(C.string, G.string)

const NumberFromString2 = make(NumberFromString, G.number)

export const MyUnion2: Compat<number | string> = union(NumberFromString2, string)

..or... something else? I don't know, any idea?

Does that mean there's no longer such a thing as a Decoder from I to A - effectively I is always unknown?

Yes it is, but please note that basically I is always unknown in the stable API too.

Will this affect the advice to use t.keyof({ a: null, b: null }) to achieve enum-like behaviour

Not sure what you mean, but the new way is

import * as D from 'io-ts/lib/Decoder'

const MyEnum = D.literal('a', 'b')

and literal is supported by Encoder, so you don't need untagged unions for that.

Does that mean anything for #373? The request there was to keep un-branded refinements as an option

The new refinement function has the following signature

export declare function refinement<A, B extends A>(
  from: Decoder<A>,
  refinement: (a: A) => a is B,
  expected: string
): Decoder<B>

where B is supposed to be different from A, while the old refinement function has the following signature

export declare function refinement<C extends Any>(
  codec: C,
  predicate: Predicate<TypeOf<C>>,
  name?: string
): RefinementC<C>

which I still consider a bad API since the predicate is not carried to the type level.

[steida]

@gcanti Will be JSON type possible?

[IMax153]

@steida Here is how I solved it using the suggestions from @gcanti above.

import * as C from 'io-ts/lib/Codec';
import * as D from 'io-ts/lib/Decoder';
import * as E from 'io-ts/lib/Encoder';
import * as G from 'io-ts/lib/Guard';

export interface Compat<A> extends D.Decoder<A>, E.Encoder<A>, G.Guard<A> {}

export const makeCompat: <A>(c: C.Codec<A>, g: G.Guard<A>) => Compat<A> = (c, g) => ({
  is: g.is,
  decode: c.decode,
  encode: c.encode,
});

export const lazy = <A>(id: string, f: () => Compat<A>): Compat<A> => {
  return makeCompat(C.lazy(id, f), G.guard.lazy(id, f));
};

export const untaggedUnion: <A extends ReadonlyArray<unknown>>(
  ...ms: { [K in keyof A]: Compat<A[K]> }
) => Compat<A[number]> = (...ms) => ({
  is: G.guard.union(...ms).is,
  decode: D.decoder.union(...ms).decode,
  encode: (a) => ms.find((m) => m.is(a)),
});

type Json = string | number | boolean | null | { [property: string]: Json } | Json[];

const Json: Compat<Json> = lazy<Json>('Json', () =>
  untaggedUnion(
    makeCompat(C.string, G.string),
    makeCompat(C.number, G.number),
    makeCompat(C.boolean, G.boolean),
    makeCompat(C.literal(null), G.literal(null)),
    makeCompat(C.record(Json), G.record(Json)),
    makeCompat(C.array(Json), G.array(Json)),
  ),
);

const json = Json.decode([1, [[['1']], { a: 1, b: false }]]);
console.log(json);
// {
//    _tag: 'Right',
//    right: [ 1, [ [ [ '1' ] ], { a: 1, b: false } ] ]
// }

[gcanti]

@IMax153 @steida the Json encoder is just the identity function

import * as C from 'io-ts/lib/Codec'
import * as D from 'io-ts/lib/Decoder'
import * as E from 'io-ts/lib/Encoder'

type Json = string | number | boolean | null | { [key: string]: Json } | Array<Json>

const JsonDecoder = D.lazy<Json>('Json', () =>
  D.union(C.string, C.number, C.boolean, C.literal(null), C.record(Json), C.array(Json))
)

const Json: C.Codec<Json> = C.make(JsonDecoder, E.id)

[mmkal]

..or... something else? I don't know, any idea?

@gcanti maybe there could be a special case for unions of codecs with encode = t.identity. If t.identity itself were branded, then you could know at the type level and at runtime that it's safe to encode with any of the sub-types. It would cover the "simple" cases where io-ts is just used for validation, which are quite common:

t.union([t.string, t.number, t.type({ myProp: t.boolean })])

☝️ that's assuming it's possible to "propagate" the identity encoder from props, i.e. t.type({ myProp: t.boolean }) satisfies the requirement of all its props being identity-encoders, so it is one too.

note that basically I is always unknown in the stable API too.

Not in the case of typeA.pipe(typeB), but it sounds like that whole flow is going to change somewhat - if so, some kind of migration might be worth noting somewhere in the docs.

[gcanti]

maybe there could be a special case for unions of codecs with encode = t.identity

@mmkal maybe, but supporting untagged unions means that the implementation of Schemable's union

declare function union<A extends ReadonlyArray<unknown>>(
  ...members: { [K in keyof A]: Encoder<A[K]> }
): Encoder<A[number]>

should work with any Encoder.

Not in the case of typeA.pipe(typeB)

I think that most of the times .pipe(...) can be replaced by parse

declare function parse<A, B>(from: Decoder<A>, parser: (a: A) => Either<string, B>): Decoder<B>

[leemhenson]

I think it would be useful to include a pipe-like fn anyway, for when you already have two decoders that you want to sequence:

const pipe = <A>(da: D.Decoder<A>) => <B>(db: D.Decoder<B>): D.Decoder<B> => ({
  decode: flow(da.decode, E.chain(db.decode)),
});

Or, I dunno, would calling that D.chain make more sense? That's what I instinctively reached for.

[leemhenson]

Another thing I miss from the new API is a name attribute on a Decoder/Encoder. I like to wrap the library's own DecodeError in my own Error subclass so I can attach more metadata, using something like this:

export const decode = <A>(
  decoder: Decoder<A>,
  errorMetadata?: Record<string, unknown>
) =>
  flow(
    decoder.decode,
    E.mapLeft(error => makeIoTsDecodeError(error, errorMetadata))
  );

I used to be able to automatically smoosh in the Type<A, O, I>.name:

makeIoTsDecodeError(error, { decoderName: decoder.name, ...errorMetadata }

But that's not possible with Decoder<A> now.

I suppose I could extend Decoder to add it back in just inside my projects but I wonder what was the reason for dropping name?

[gcanti]

I think it would be useful to include a pipe-like fn anyway, for when you already have two decoders that you want to sequence

example?

I wonder what was the reason for dropping name?

Because it makes the APIs unnecessarily complicated, IMO the error messages are readable even without the names (and you can always use withExpected if you don't like the default)

[leemhenson]

example?

I know D.parse exists for effectively chaining A => B on the end of an earlier decoder, but you have to return Either<string, B>. If I already have a decoder that provides A => B then it's easiest for me to just compose them together. I suppose there's some tension there because decode is always taking u: unknown as opposed to a hard requirement on A, which D.parse does give you. 🤷

In some ways this feels similar to the discussion we had recently about re-adding the second type parameter to Encoder. It almost feels to me like we should have Decoder<Output, Input = unknown>, and D.chain(dua: Decoder<A>, dab: Decoder<B, A>) => Decoder<B>.

withExpected

Yeah I think i need to experiment with that a bit - I guess I would use it to replace the text inside a leaf?

[mmkal]

@gcanti re D.parse vs the old Type.prototype.pipe. What would you recommend for a base64-json decoder. i.e. one that decodes base 64, parses the decoded string as JSON, then validates the json using io-ts. An example use case is handling kinesis events, which trigger lambdas with base64 payloads. With the current io-ts, it's possible to use a combinator like:

const MyEvent = kinesisEvent(t.type({ foo: t.string }))

The kinesisEvent combinator can ensure:

• the input value looks like { records: Array<{ recordId: string; data: string }> }
• each records[*].data is a string
• each string is valid base 64
• when decoded, the strings are valid json
• when json-parsed, the decoded strings have structure { foo: string }

Is this possible with the new D.parse? Would you recommend @leemhenson's method - if so it would be great if there were a first-class helper for it to avoid many implementations in downstream projects that might miss edge cases.

---

Another question, since this issue title is "High-level explanation of API changes". Could you give a recommendation for users who rely on myType.props for interface and partial types, and myType.types for union and intersection types in the stable API? From comments like this it sounds like there isn't a replacement yet, for reflection-like functionality. Use cases include UI-generation, dynamic codec manipulation, etc. In this comment you mentioned development of the stable API is frozen. Does this mean the functionality is going to be replaced by something else?

[gcanti]

Is this possible with the new D.parse?

Yes, but there are many different ways to get the final result so I guess it really depends on your coding style ("everything is a decoder" or "I want to lift by business / parsing logic only once"?).

For example

import * as E from 'fp-ts/lib/Either'
import { Json } from 'io-ts/lib/JsonEncoder'
import * as D from 'io-ts/lib/Decoder'
import { flow } from 'fp-ts/lib/function'

// > decodes base 64
declare function decodeBase64(s: string): E.Either<string, string>

// > parses the decoded string as JSON
declare function parseJSON(s: string): E.Either<string, Json>

// > then validates the json using io-ts
declare function decodeItem(json: Json): E.Either<string, { foo: string }>

const parser = flow(decodeBase64, E.chain(parseJSON), E.chain(decodeItem))

export const X = D.parse(D.string, parser)

it sounds like there isn't a replacement yet

Actually one of the goals of my rewrite was to get rid of those meta infos (at the type level)

[gcanti]

@leemhenson @mmkal given the good results in https://github.com/gcanti/io-ts/issues/478 I'm going to make the following breaking changes:

• Decoder
  • change DecoderError
  • remove never
  • make parse pipeable and change its parser argument
• Guard
  • remove never
• Schemable
  • make intersections pipeables
  • make refinements pipeables

parse

from

declare export function parse<A, B>(from: Decoder<A>, parser: (a: A) => Either<string, B>): Decoder<B>

to

declare export function parse<A, B>(parser: (a: A) => E.Either<DecodeError, B>): (from: Decoder<A>) => Decoder<B>

Pros:

• more general (DecodeError instead of string)
• pipeable
• should accomodate the pipe use case

import { pipe } from 'fp-ts/lib/pipeable'
import * as D from '../src/Decoder2'
import { Json } from '../src/JsonEncoder'

// > decodes base 64
declare const Base64: D.Decoder<string>

// > parses the decoded string as JSON
declare const Json: D.Decoder<Json>

// > then validates the json using io-ts
declare const Item: D.Decoder<{ foo: string }>

export const X = pipe(D.string, D.parse(Base64.decode), D.parse(Json.decode), D.parse(Item.decode))

[leemhenson]

Are you changing the signatue of

export interface Decoder<A> {
  readonly decode: (u: unknown) => Either<DecodeError, A>
}

to:

export interface Decoder<A, B = unknown> {
  readonly decode: (u: B) => Either<DecodeError, A>
}

?

Otherwise Base64 and Json are both going to have to repetitively test inside their decode implementations whether u is actually a string before performing string-based operations. Micro-optimizations maybe, but as you combine more and more parsers together to operate on larger and larger structures, it could start to add up.

[gcanti]

@leemhenson isn't what happens in your proposal too?

const pipe = <A>(da: D.Decoder<A>) => <B>(db: D.Decoder<B>): D.Decoder<B> => ({
  decode: flow(da.decode, E.chain(db.decode)),
});

[leemhenson]

Yes, I never said mine was optimal 😅 . I'm just re-raising the point I made earlier:

In some ways this feels similar to the discussion we had recently about re-adding the second type parameter to Encoder. It almost feels to me like we should have Decoder<Output, Input = unknown>, and D.chain(dua: Decoder<A>, dab: Decoder<B, A>) => Decoder<B>.

If we did that, then the piped decoders wouldn't need to keep checking the same things over and over.

[leemhenson]

Why is that in bold? Emphasis not mine! 😬

[gcanti]

@leemhenson maybe it's just a bias of mine but I consider a "proper decoder" an arrow that goes from unknown to some type A

unknown -> M<A>

for some effect M, because otherwise it's just a "normal" kleisli arrow

A -> M<B>

and I use chain to compose those.

So personally I would model my pipeline starting from normal kleisli arrows and then I would define a suitable decoder based on the use case at hand.

// kleisli arrows in my domain

declare function decodeBase64(s: string): E.Either<string, string>

declare function parseJSON(s: string): E.Either<string, Json>

declare function decodeItem(json: Json): E.Either<string, { foo: string }>

// I can compose them as usual using the `Monad` instance of `Either`

const decode = flow(decodeBase64, E.chain(parseJSON), E.chain(decodeItem))

// and then define my decoder

export const MyDecoder = pipe(
  D.string,
  D.parse((s) =>
    pipe(
      decode(s),
      E.mapLeft((e) => D.error(s, e))
    )
  )
)

Alternatively I could have already defined some decoders

declare const Base64: D.Decoder<string>

declare const Json: D.Decoder<Json>

declare const Item: D.Decoder<{ foo: string }>

if this is the case, again I can compose them via parse

// and I can compose them via parse

export const MyDecoder2 = pipe(Base64, D.parse(Json.decode), D.parse(Item.decode))

// or even this if I want micro optimizations
export const MyDecoder3 = pipe(
  Base64,
  D.parse((s) =>
    pipe(
      parseJSON(s),
      E.mapLeft((e) => D.error(s, e))
    )
  ),
  D.parse(Item.decode)
)

Do we really need something more? Genuine question, I'm open to suggestions if you think there's an ergonomic issue with the current APIs.

In the end if you can define a

export interface Decoder<A, B> {
  readonly decode: (a: A) => Either<DecodeError, B>
}

then you can just define f = (a: A) => Either<DecodeError, B> and use parse to compose f with a Decoder<A>

[leemhenson]

It's not a major issue, just a niggle I keep encountering because I have scenarios like these:

• desire to decode from some wire representation into a branded type, for simplicity let's say Int
• wire types might be numeric or a string representation of a number, e.g. 3 or "3"
• an Int might also be further branded into FooId or Cents or something, and that logic might include doing some bounds checking

So in this case I would like to have primitive decoders:

intFromNumber: Decoder<Int, number>
intFromString: Decoder<Int, string>
fooIdFromInt: Decoder<FooId, Int>
centsFromInt: Decoder<Cents, Int>

then I can compose them together to make complex ones:

pipe(
  stringFromUnknown,
  intFromString,
  fooIdFromInt,
) // => Decoder<FooId, unknown>

I could wrap all that logic up using Decoder<FooId> as it is today but if I have useful chains of decoders that I want to reuse it always ends up feeling like it would be more elegant to make the composition at the Decoder level.

But, hey, that's just me. I might just be using the wrong tool for the job. 🤷

[gcanti]

it always ends up feeling like it would be more elegant to make the composition at the Decoder level

Well, while I love io-ts, as a user I would try (as far as possible) to not leak an implementation detail (i.e. which library I'm using to validate / decode at the border). In my app domain I would prefer to define

intFromNumber: number -> Either<MyDomainError, Int>
intFromString: string -> Either<MyDomainError, Int>
fooIdFromInt: Int -> Either<MyDomainError, FooId>
centsFromInt: Int -> Either<MyDomainError, Cents>

that will be future-proof even if a I replace io-ts with another solution.

In elm-ts I removed the hard dependency on io-ts by just requiring a kleisli arrow. I will do the same for fp-ts-routing in the next major release.

But that's just a point of view, yours is sensible too.

Let me just think more about all of this...

[leemhenson]

... as a user I would try (as far as possible) to not leak an implementation detail (i.e. which library I'm using to validate / decode at the border). In my app domain I would prefer to define ...

Yeah I'm only talking about composition of Decoders as a means to construct the larger Decoders that I use at the app boundary to convert unknown => Either<DecodeError, SomeComplexNestedProduct>. I don't see the Decoder in the rest of the codebase.

[gcanti]

In the end if you can define a

export interface Decoder<A, B> { readonly decode: (a: A) => Either<DecodeError, B> }

then you can just define f = (a: A) => Either<DecodeError, B>

However the converse is also true, so my POV is actually biased and without noticeable substance, @leemhenson I'll reconsider my position.

[gcanti]

@leemhenson while experimenting with kleisli arrows looks like I found something more general than DecoderT (see the Kleisli module):

interface Kleisli<M extends URIS2, I, E, A> {
  readonly decode: (i: I) => Kind2<M, E, A>
}

for which I can define a compose operation

const compose = <M extends URIS2, E>(M: Monad2C<M, E>) => <A, B>(ab: Kleisli<M, A, E, B>) => <I>(
  ia: Kleisli<M, I, E, A>
): Kleisli<M, I, E, B> => ({
  decode: (i) => M.chain(ia.decode(i), ab.decode)
})

Then from Kleisli I can derive KleisliDecoder

interface KleisliDecoder<I, A> extends K.Kleisli<E.URI, I, DecodeError, A> {}`

and from KleisliDecoder I can derive our old Decoder

interface Decoder<A> extends KD.KleisliDecoder<unknown, A> {}

Example

import { pipe } from 'fp-ts/lib/pipeable'
import * as D from '../src/Decoder'
import * as KD from '../src/KleisliDecoder'

interface IntBrand {
  readonly Int: unique symbol
}
type Int = number & IntBrand
interface CentsBrand {
  readonly Cents: unique symbol
}
type Cents = number & CentsBrand

declare const IntFromString: KD.KleisliDecoder<string, Int>
declare const CentsFromInt: KD.KleisliDecoder<Int, Cents>

// const result: D.Decoder<Cents>
export const result = pipe(
  D.string, 
  D.compose(IntFromString), 
  D.compose(CentsFromInt)
)

[leemhenson]

Love it.

[gcanti]

KleisliDecoder is quite interesting in that its input type is fine grained (i.e. not a generic Record<string, I>) and depends on the fields passed in

/*
const kdecoder: KD.KleisliDecoder<{
    name: unknown;
    age: string;
    cents: Int;
}, {
    name: string;
    age: Int;
    cents: Cents;
}>
*/
export const kdecoder = KD.type({
  name: D.string,
  age: IntFromString,
  cents: CentsFromInt
})

EDIT: same for tuple, etc...

// const kdecoder2: KD.KleisliDecoder<[unknown, string, Int], [string, Int, Cents]>
export const kdecoder2 = KD.tuple(D.string, IntFromString, CentsFromInt)

[gcanti]

v2.2.7 released https://github.com/gcanti/io-ts/releases/tag/2.2.7

[osdiab]

as someone not super well versed in category theory, when does it make sense to use KleisliDecoder functions? I get the general concept of Kleisli arrows and can figure it out when it gets stable but I predict difficulty explaining these things to my team lol

is it mostly that Decoder functions assume unknown as the input type/are generally implemented in terms of the KleisliDecoder functions, but KleisliDecoder functions also allow control over the input type (in a sense closer to the original io-ts Type generic params)? and how does that relate to Kleisli arrows? sorry for the question but i'm kinda out of my depth