Is there a way to write a codec that fails on additional properties?

[Mousaka]

Do you want to request a feature or report a bug? Question: Is there a way to write a codec that fails on additional properties?

const FruitOrAnimal = t.union([
  t.strict({ fruit: t.string }),
  t.strict({ animal: t.string })
]);

console.log(FruitOrAnimal.decode({ fruit: "banana", animal: "cat" }).isRight());
//prints  'true' but would like it to be 'false'.

Would like a decoder that fails this input type and interpret it as an "impossible state". An input with either fruit or animal is fine but never both.

What is the current behavior? https://codesandbox.io/embed/8483r1pwq9

Which versions of io-ts, and which browser and OS are affected by this issue? Did this work in previous versions of io-ts? 1.8.5

[tlrobinson]

A horrible workaround I found but probably wouldn't recommend is to do a deep comparison of the original and decoded objects, since excess properties are stripped when decoding:

import { diffString } from "json-diff";

const original = { fruit: "banana", animal: "cat" };
const decoded = FruitOrAnimal.decode(original).value;
console.log(diffString(original, decoded));

logs:

 {
-  animal: "cat"
 }

I think the issue I just filed #335 is a duplicate of this one. I'll close mine.

[goooseman]

Are there any plans or updates about this ticket?

I'm stuck a little bit, because I've implemented io-ts validations in the project, but they are useless without strict type validations.

I can help to make a PR, no problem, but let's discuss how can I implement this feature.

@gcanti what do you think about this issue?

[goooseman]

My proposal:

Add a new codec: identical (maybe we need to find a better name)

Which will fail on any additional properties.

Is it possible in the current library implementation?

[gcanti]

but they are useless without strict type validations

@goooseman Why? Decoders main duty is trying to deserialize a (possibly larger) data structure into another, why additional fields should be a problem (if they are stripped anyway)?

[goooseman]

I’m using this library to validate the format of the API response, which I do not control.

So I’ve made the io-ts type for the API response and I expect the validation to fail, if some property have appeared in the response, which is not in the validation type.

Additionally, I’m using this io-told validation in the test, as a development utility to write typings. And then I need it to fail on some properties, which I did not include in the type.

Lastly, it is the original typescript functionality. If you attach type { foo: string } to a variable, you can not add additional properties.

[giogonzo]

Lastly, it is the original typescript functionality. If you attach type { foo: string } to a variable, you can not add additional properties.

Just pointing out that this is not true in general in TS, being it a "structural" type system. There are a few special cases where excess properties are checked, namely object literal assignments:

type A = { foo: string }
const a1: A = { foo: 'foo', bar: 1 } // fails because of the excess prop `bar`

const a2 = { foo: 'foo', bar: 1 }
const a3: A = a2 // doesn't fail

[goooseman]

Yep, that’s exactly what I’ve meant.

But I do not want to change the defaults of course. But propose just to add a new type specially for this use case.

There are only three questions:

1. Can I create this type as a pull request to this library?
2. How to call it?
3. Is it possible to implement this kind of decoder?

[gcanti]

Can I create this type as a pull request to this library?

No, unless there are type safety concerns (that's why I asked why, I understand what you want to achieve). Note that from a type safety POV we don't even need to strip additional fields.

Is it possible to implement this kind of decoder?

Yes it is (with some caveats).

A sketch:

import * as t from 'io-ts' // v1.10.4

function getExcessProps(props: t.Props, r: Record<string, unknown>): Array<string> {
  const ex: Array<string> = []
  for (const k of Object.keys(r)) {
    if (!props.hasOwnProperty(k)) {
      ex.push(k)
    }
  }
  return ex
}

export function excess<C extends t.InterfaceType<t.Props>>(codec: C): C {
  const r = new t.InterfaceType(
    codec.name,
    codec.is,
    (i, c) =>
      t.UnknownRecord.validate(i, c).chain(r => {
        const ex = getExcessProps(codec.props, r)
        return ex.length > 0
          ? t.failure(
              i,
              c,
              `Invalid value ${JSON.stringify(i)} supplied to : ${
                codec.name
              }, excess properties: ${JSON.stringify(ex)}`
            )
          : codec.validate(i, c)
      }),
    codec.encode,
    codec.props
  )
  return r as any
}

const C = excess(
  t.type({
    a: t.string
  })
)

import { PathReporter } from 'io-ts/lib/PathReporter'

console.log(PathReporter.report(C.decode({ a: 'a', b: 1 })))
// [ 'Invalid value {"a":"a","b":1} supplied to : { a: string }, excess properties: ["b"]' ]

Caveats

Looks like it wouldn't play well with intersections

const I = t.intersection([C, t.type({ b: t.number })])

console.log(PathReporter.report(I.decode({ a: 'a', b: 1 })))
// [ 'Invalid value {"a":"a","b":1} supplied to : { a: string }, excess properties: ["b"]' ]

[goooseman]

Thanks for the help.

I've published the library to fail on additional properties as an additional package. @mousaka Take a look

[josejulio]

Regarding this subject, I'm not worried about additional properties, but more worried about possible typos made to optional properties. Consider the following:

const ExternalServiceInfoCodec = t.exact(
  t.intersection([
    t.interface({
      name: t.string
    }),
    t.partial({
      version: t.string,
      url: t.string
    })
], 'ExternalServiceInfo'));

const test = { 
  name: "Hello world",
  versionx: "1.3"
};

ExternalServiceInfoCodec.decode(test);
// or 
if (!ExternalServiceInfoCodec.is(test)) {
  console.error('wrong');
}

This will succeed and we won't immediately notice that we had a typo in versionx (instead of version). Sorry for hijacking this thread, but is somewhat similar.

Is there any way for io-ts to help in this regard?

[Mousaka]

Thanks for the help.

I've published the library to fail on additional properties as an additional package. @Mousaka Take a look

Nice!

[MastroLindus]

I am also interested in this ticket, and I am not a big fan of adding an additional package/dependency only for fixing it. I use io-ts to validate api calls from client to the server (on the server I want to ensure that the client calls my API exactly with the types allowed, and not with any additional/different parameters). Because decode simply strips additional parameters, I cannot detect easily if those extra parameters were present. Honestly I find a bit deceiving that if you mark a type as exact or an intersection of exact types, decode still allows extra fields to be used without adding any error.

[giogonzo]

One of the key design choices of io-ts is (I believe) mapping 1:1 with the TS type system, and this to me looks also one of the main strengths compared to other solutions for parsing/validation. And, as we all know, "excess property checks" is not a feature of the TS types system (if not in some specific instances)

At the same time I can understand the desire to have a drop in solution compatible with io-ts to obtain this behavior, and I think the best starting point could be to move @gcanti example from https://github.com/gcanti/io-ts/issues/322#issuecomment-513170377 into io-ts-types

[MastroLindus]

I would in general agree. However given that Io-ts provides exact/strict types, I think the current behaviour of decoding an exact type even when it has additional properties is deceiving. Either exact/strict types should also be part of io-ts-types, or (as I hope, since they are super useful) if they stay in io-ts the decode should allow you to detect if there are extra properties without relying on manually deep comparing the original object and the "right" part of the decoded one

[giogonzo]

I did a POC of "excess" in io-ts-types here: https://github.com/gcanti/io-ts-types/pull/114

It is mostly a porting of https://github.com/gcanti/io-ts/issues/322#issuecomment-513170377, but, as requested in #378, also applies the same "excess" check in the .is method.

I've discussed that PR offline with @gcanti and I'm closing it, the reason being we prefer not to make it "official" for the moment, since "excess" doesn't work well with all the codecs, t.intersection for example, and also because the exact behavior desired by different people is varying (e.g. checking for "excess" in .is or not)

It think this issue could now be closed since there are already many linked solutions available: please refer to https://github.com/goooseman/io-ts-excess by @goooseman or to my POC here if you want the .is excess check

[mnn]

It think this issue could now be closed since there are already many linked solutions available

If I am reading correctly, neither solution works with intersection which is unfortunate, because we use intersection literally in every case. After upgrading from 1.4 we hit this unexpected regression. Should I create a new ticket (specifically mentioning intersection, I don't see any opened ticket regarding this issue), since you want to close this one?

[giogonzo]

~@mnn you can always apply excess to the components of the intersection right?~

EDIT: disregard, it obviously doesn't work: it would fail at the very first component, incorrectly

[mnn]

Any progress on this? Just today I encountered a bug in our newer project which would have been caught ages ago if exact was checking invalid fields. Time to downgrade I guess... :disappointed:

[jloleysens]

I think it would be good to improve documentation on this. There are a number of users that expect const a: { b: number } = { b: 1 /* nothing else */ } behaviour, somewhat reasonably as this is also how TS behaves which means this issue will probably surface again. Explaining that t.type and friends implement structural typechecking (as TS does) and its implications in the README.md would help users a lot.

I think this is a usability issue for io-ts that is unfortunate and it would definitely be good to help users to a solution. If the closest/best solution does not play well with other codecs it should still be provided (somehow) with a caveat. What can one reasonably expect if you want to disallow additional props and intersect with another structually checked type? That is in the runtime domain. I believe it would be best to be informed of this behaviour and be empowered to make the decision for myself on an as-needed basis.

[noe132]

So if there is exact type here, I think by doing something similar to what exact type does, instead of striping additional properties, make a slightly adjustment to the striping function and let it do the checking work, if any property was stripped, then we know for sure there is additional properties.

I wrote a small excess type that works exactly as exact type did, but instead of stripping properties, it reports error if any additional properties were found. It looks woking with intersection.

I've test this a little bit, and I havn't came with any problem.

https://codesandbox.io/s/inspiring-wright-u4wk9 open the codesandbox console to check output

import * as t from 'io-ts'
import { excess } from './excess'

const codec = excess(t.intersection([
  t.type({
    a: t.string.
  }),
  t.partial({
    b: t.string.
  }),
]))

type Type = t.TypeOf<typeof Codec>
// Type is equivalent to
interface Type {
  a: string
  b?: string
}

codec.decode({ a: 'apple' }) // right
codec.decode({ a: 'apple', b: 'banana' }) // right
codec.decode({ a: 'apple', b: 'banana', c: 'coconut' }) // left

// excess.ts
import * as t from 'io-ts'
import { either, Either, isRight, left, right, Right } from 'fp-ts/lib/Either'

const getIsCodec = <T extends t.Any>(tag: string) => (codec: t.Any): codec is T => (codec as any)._tag === tag
const isInterfaceCodec = getIsCodec<t.InterfaceType<t.Props>>('InterfaceType')
const isPartialCodec = getIsCodec<t.PartialType<t.Props>>('PartialType')

const getProps = (codec: t.HasProps): t.Props => {
  switch (codec._tag) {
    case 'RefinementType':
    case 'ReadonlyType':
      return getProps(codec.type)
    case 'InterfaceType':
    case 'StrictType':
    case 'PartialType':
      return codec.props
    case 'IntersectionType':
      return codec.types.reduce<t.Props>((props, type) => Object.assign(props, getProps(type)), {})
  }
}

const getNameFromProps = (props: t.Props): string => Object.keys(props)
  .map((k) => `${k}: ${props[k].name}`)
  .join(', ')

const getPartialTypeName = (inner: string): string => `Partial<${inner}>`

const getExcessTypeName = (codec: t.Any): string => {
  if (isInterfaceCodec(codec)) {
    return `{| ${getNameFromProps(codec.props)} |}`
  } if (isPartialCodec(codec)) {
    return getPartialTypeName(`{| ${getNameFromProps(codec.props)} |}`)
  }
  return `Excess<${codec.name}>`
}

const stripKeys = <T = any>(o: T, props: t.Props): Either<Array<string>, T> => {
  const keys = Object.getOwnPropertyNames(o)
  const propsKeys = Object.getOwnPropertyNames(props)

  propsKeys.forEach((pk) => {
    const index = keys.indexOf(pk)
    if (index !== -1) {
      keys.splice(index, 1)
    }
  })

  return keys.length
    ? left(keys)
    : right(o)
}

export const excess = <C extends t.HasProps>(codec: C, name: string = getExcessTypeName(codec)): ExcessType<C> => {
  const props: t.Props = getProps(codec)
  return new ExcessType<C>(
    name,
    (u): u is C => isRight(stripKeys(u, props)) && codec.is(u),
    (u, c) => either.chain(
      t.UnknownRecord.validate(u, c),
      () => either.chain(
        codec.validate(u, c),
        (a) => either.mapLeft(
          stripKeys<C>(a, props),
          (keys) => keys.map((k) => ({
            value: a[k],
            context: c,
            message: `excess key "${k}" found`,
          })),
        ),
      ),
    ),
    (a) => codec.encode((stripKeys(a, props) as Right<any>).right),
    codec,
  )
}

export class ExcessType<C extends t.Any, A = C['_A'], O = A, I = unknown> extends t.Type<A, O, I> {
  public readonly _tag: 'ExcessType' = 'ExcessType'
  public constructor(
    name: string,
    is: ExcessType<C, A, O, I>['is'],
    validate: ExcessType<C, A, O, I>['validate'],
    encode: ExcessType<C, A, O, I>['encode'],
    public readonly type: C,
  ) {
    super(name, is, validate, encode)
  }
}

[rjhilgefort]

@noe123 That type worked out amazingly! Thank you so much for posting it here! 🙏🙏🙏

[mixedCase]

@gcanti Just to mention an experience, not sure if a codec that fails on additional properties is the answer since the true failure here is my own, but:

I just made a codec which is essentially:

t.union([
  t.type({ thing: t.number }),
  t.type({ thing: t.number, anotherThing: t.string })
])

Given a payload of { thing: 123, anotherThing: 324 }, it passes the first type of the union. However I made the mistake of differentiating the two by using "anotherThing" in payload, which TypeScript happily treated as the second case, because that's the only one with an anotherThing field.

Granted, once looking in more deeply the solution I came up with is not to use t.union and then discriminate for cases like this, but to provide different code paths for each codec since the in check means (in practice) different things to the type-checker compared to runtime. But something that allows me to specify that if the first case has other fields then it's no longer that case would've saved me an hour of debugging.

[lirbank]

I'm also interested in this functionality. I'm working on a dev utility (bson-schema-to-typescript) which reads a JSON configuration file.

I kind of like the idea to let the user know if there are any configuration errors, being it using a wrong type for a config option (io-ts) or misspelling a configuration option (excess property check).

In my case, the configuration JSON object is flat so it is easy to check for excess properties, but it'd be cool if io-ts could handle it.

Just sharing another use case. Love the lib though!

[badeball]

Just chiming in to tell you about my use case and why I would like to see excess property check. We're using io-ts to parse and decode data tables in feature files (Cucumber). Some of our tables contain optional properties, but failure to spell these properties correctly will naturally lead to unexpected behavior. It would be beneficial to us if such would lead to instant feedback.