io-ts
io-ts copied to clipboard
Published
20 hours ago •
gcanti
gcanti
How to decode nested json involving string union literal types
I want to decode following raw json to the Parameter model. I have tried various variations but failed.
Does io-ts support following use case
// input json to be decoded to Parameter class
const raw: unknown = {
IntKey: {
keyName: 'epoch',
values: [1, 2, 3]
}
}
// domain models
type IntKey = {
KeyTag: 'IntKey'
KeyType: number
}
type StringKey = {
KeyTag: 'StringKey'
KeyType: string
}
type Key = IntKey | StringKey
class Parameter<T extends Key> {
constructor(
readonly keyName: string,
readonly keyTag: T['KeyTag'],
readonly values: T['KeyType'][]
) {}
}
Answering my own question, I have come up with following code which does exactly what I mentioned in original issue.
Is there any simplification possible?
import * as E from 'fp-ts/lib/Either'
import { pipe } from 'fp-ts/lib/function'
import * as t from 'io-ts'
class Parameter<T extends Key> {
constructor(
readonly keyName: string,
readonly keyTag: T['keyTag'],
readonly values: T['keyType'][]
) {}
}
const ParamBodyDecoder = <T>(valuesDec: t.Type<T, unknown>) =>
t.type({
keyName: t.string,
values: t.array(valuesDec)
})
const MakeKey = <KType extends t.Mixed>(kType: KType) => <KTag extends string>(kTag: KTag) =>
t.type({
keyTag: t.literal(kTag),
keyType: kType,
paramDecoder: ParamBodyDecoder(kType)
})
const IntKey = MakeKey(t.number)('IntKey')
const StringKey = MakeKey(t.string)('StringKey')
type IntKey = t.TypeOf<typeof IntKey>
type StringKey = t.TypeOf<typeof StringKey>
type Key = IntKey | StringKey
const Keys = {
IntKey: IntKey,
StringKey: StringKey
}
const KeyTag = t.keyof(Keys)
type KeyTag = t.TypeOf<typeof KeyTag>
type ParameterJson<T extends Key> = {
keyName: string
values: T['keyType'][]
}
type ParameterJsonResult<T extends Key> = E.Either<t.Errors, ParameterJson<T>>
const decodeKeyTag = (record: Record<string, unknown>) => KeyTag.decode(Object.keys(record)[0])
const decodeParamBody = <T extends Key>(
keyTag: KeyTag,
record: Record<string, unknown>
): ParameterJsonResult<T> => Keys[keyTag].props.paramDecoder.decode(record[keyTag])
const toParameter = <T extends Key>(keyTag: KeyTag, rawParam: ParameterJson<T>) =>
new Parameter(rawParam.keyName, keyTag, rawParam.values)
const decodeParameter = <T extends Key>(input: unknown): t.Validation<Parameter<T>> =>
pipe(
t.UnknownRecord.decode(input),
E.chain((record) =>
pipe(
decodeKeyTag(record),
E.chain((keyTag) =>
pipe(
decodeParamBody<T>(keyTag, record),
E.map((body) => toParameter<T>(keyTag, body))
)
)
)
)
)
test('decode parameter', () => {
// input json to be decoded to Parameter class
const raw: unknown = {
IntKey: {
keyName: 'epoch',
values: [1, 2, 3]
}
}
console.log(decodeParameter(raw))
})
// ======== OUTPUT ========
{
_tag: 'Right',
right: Parameter { keyName: 'epoch', keyTag: 'IntKey', values: [ 1, 2, 3 ] }
}
Is there any simplification possible?
@kpritam I would define a decoder for the raw input (I'm guessing):
const decoder = t.partial({
IntKey: t.type({
keyName: t.string,
values: t.array(t.number)
}),
StringKey: t.type({
keyName: t.string,
values: t.array(t.string)
})
})
and then map the result
const decodeParameters = (input: unknown): t.Validation<Array<Parameter<Key>>> =>
pipe(
decoder.decode(input),
E.map((a) => {
const out: Array<Parameter<Key>> = []
if (a.IntKey !== undefined) {
out.push(new Parameter(a.IntKey.keyName, 'IntKey', a.IntKey.values))
}
if (a.StringKey !== undefined) {
out.push(new Parameter(a.StringKey.keyName, 'StringKey', a.StringKey.values))
}
return out
})
)
@gcanti thanks for the response.
This was the minimal example, but in real case, we have more than 15 such a keys (IntKey, StringKey, BooleanKey, IntArrayKey etc.).
We are also using independent static types like IntKey, StringKey etc. generated from io-ts runtime types in different places in our code bases.
Hence I am first decoding index key and based on that using appropriate corresponding decoder to decode rest of the body. In order to do that, I had to keep all the keys in record so that I can retrieve correct decoder based on provided key. I am not able to get rid this record somehow.
const Keys = {
IntKey: IntKey,
StringKey: StringKey
}
I see, but what I meant is that without the schema of the raw input is hard to say something and help you out.
A single example
// input json to be decoded to Parameter class
const raw: unknown = {
IntKey: {
keyName: 'epoch',
values: [1, 2, 3]
}
}
is not enough.
Here you are picking the first key only
const decodeKeyTag = (record: Record<string, unknown>) => KeyTag.decode(Object.keys(record)[0])
does it mean that the input schema is a union instead?
const decoder = t.union([
t.type({
IntKey: t.type({
keyName: t.string,
values: t.array(t.number)
})
}),
t.type({
StringKey: t.type({
keyName: t.string,
values: t.array(t.string)
})
}),
... etc ...
])
I can just make arbitrary guesses
Input schema is this :
{ paramSet: Parameter<Key>[] }
Note: StructKey is recursive which is nothing but { paramSet: Parameter<Key>[] }
Complete sample json file looks like this:
{
paramSet: [
{
LongKey: {
keyName: 'LongKey',
values: [50, 60],
units: 'NoUnits'
}
},
{
RaDecKey: {
keyName: 'RaDecKey',
values: [
{
ra: 7.3,
dec: 12.1
}
],
units: 'NoUnits'
}
},
{
FloatKey: {
keyName: 'FloatKey',
values: [90, 100],
units: 'NoUnits'
}
},
{
StructKey: {
keyName: 'StructKey',
values: [
{
paramSet: [
{
BooleanKey: {
keyName: 'BooleanKey',
values: [true, false],
units: 'NoUnits'
}
},
{
ByteKey: {
keyName: 'ByteKey',
values: [10, 20],
units: 'NoUnits'
}
}
]
}
],
units: 'NoUnits'
}
},
{
IntArrayKey: {
keyName: 'IntArrayKey',
values: [[7, 8]],
units: 'NoUnits'
}
},
{
IntMatrixKey: {
keyName: 'IntMatrix',
values: [
[
[12, 13],
[14, 15]
]
],
units: 'NoUnits'
}
},
{
LongArrayKey: {
keyName: 'LongArrayKey',
values: [[5, 6]],
units: 'NoUnits'
}
},
{
ShortKey: {
keyName: 'ShortKey',
values: [30, 40],
units: 'NoUnits'
}
},
{
LongMatrixKey: {
keyName: 'LongMatrix',
values: [
[
[8, 9],
[10, 11]
]
],
units: 'NoUnits'
}
},
{
ByteMatrixKey: {
keyName: 'ByteMatrix',
values: [
[
[1, 2],
[3, 4]
]
],
units: 'NoUnits'
}
},
{
ShortArrayKey: {
keyName: 'ShortArrayKey',
values: [[3, 4]],
units: 'NoUnits'
}
},
{
CoordKey: {
keyName: 'CoordKey',
values: [
{
_type: 'EqCoord',
tag: 'BASE',
ra: 659912250000,
dec: -109892300000,
frame: 'ICRS',
catalogName: 'none',
pm: {
pmx: 0.5,
pmy: 2.33
}
},
{
_type: 'SolarSystemCoord',
tag: 'BASE',
body: 'Venus'
},
{
_type: 'MinorPlanetCoord',
tag: 'GUIDER1',
epoch: 2000,
inclination: 324000000000,
longAscendingNode: 7200000000,
argOfPerihelion: 360000000000,
meanDistance: 1.4,
eccentricity: 0.234,
meanAnomaly: 792000000000
},
{
_type: 'CometCoord',
tag: 'BASE',
epochOfPerihelion: 2000,
inclination: 324000000000,
longAscendingNode: 7200000000,
argOfPerihelion: 360000000000,
perihelionDistance: 1.4,
eccentricity: 0.234
},
{
_type: 'AltAzCoord',
tag: 'BASE',
alt: 1083600000000,
az: 153000000000
}
],
units: 'NoUnits'
}
},
{
ByteKey: {
keyName: 'ByteKey',
values: [10, 20],
units: 'NoUnits'
}
},
{
DoubleKey: {
keyName: 'DoubleKey',
values: [110, 120],
units: 'NoUnits'
}
},
{
DoubleArrayKey: {
keyName: 'DoubleArrayKey',
values: [[11, 12]],
units: 'NoUnits'
}
},
{
DoubleMatrixKey: {
keyName: 'DoubleMatrix',
values: [
[
[20, 21],
[22, 23]
]
],
units: 'NoUnits'
}
},
{
ByteArrayKey: {
keyName: 'ByteArrayKey',
values: [[1, 2]],
units: 'NoUnits'
}
},
{
CharKey: {
keyName: 'CharKey',
values: ['65', '66'], // fixme: CharKey comes as number from scala
units: 'NoUnits'
}
},
{
IntKey: {
keyName: 'IntKey',
values: [70, 80],
units: 'NoUnits'
}
},
{
UTCTimeKey: {
keyName: 'UTCTimeKey',
values: ['1970-01-01T00:00:00Z', '2017-09-04T19:00:00.123456789Z'],
units: 'second'
}
},
{
ChoiceKey: {
keyName: 'ChoiceKey',
values: ['First', 'Second'],
units: 'NoUnits'
}
},
{
FloatArrayKey: {
keyName: 'FloatArrayKey',
values: [[9, 10]],
units: 'NoUnits'
}
},
{
StringKey: {
keyName: 'StringKey',
values: ['Str1', 'Str2'],
units: 'NoUnits'
}
},
{
BooleanKey: {
keyName: 'BooleanKey',
values: [true, false],
units: 'NoUnits'
}
},
{
TAITimeKey: {
keyName: 'TAITimeKey',
values: ['1970-01-01T00:00:00Z', '2017-09-04T19:00:00.123456789Z'],
units: 'second'
}
},
{
ShortMatrixKey: {
keyName: 'ShortMatrix',
values: [
[
[4, 5],
[6, 7]
]
],
units: 'NoUnits'
}
},
{
FloatMatrixKey: {
keyName: 'FloatMatrix',
values: [
[
[16, 17],
[18, 19]
]
],
units: 'NoUnits'
}
}
]
}
So the input schema is something along the lines of
// ---------------------------
// input schema
// ---------------------------
type InputKey<L extends string, V> = Record<L, { keyName: L; values: Array<V> }>
interface LongKey extends InputKey<'LongKey', number> {}
interface RaDecKey
extends InputKey<
'RaDecKey',
{
ra: number
dec: number
}
> {}
interface FloatKey extends InputKey<'FloatKey', number> {}
interface BooleanKey extends InputKey<'BooleanKey', boolean> {}
interface ByteKey extends InputKey<'ByteKey', number> {}
interface StructKey extends InputKey<'StructKey', Input> {}
type Param = LongKey | RaDecKey | FloatKey | StructKey | BooleanKey | ByteKey // more members...
interface Input {
paramSet: Array<Param>
}
and this is a one-to-one translation to decoders
// ---------------------------
// input decoders
// ---------------------------
const inputKey = <L extends string, V>(literal: L, values: D.Decoder<unknown, V>): D.Decoder<unknown, InputKey<L, V>> =>
D.type({
[literal]: D.type({
keyName: D.literal(literal),
values: D.array(values)
})
}) as any
const LongKey: D.Decoder<unknown, LongKey> = inputKey('LongKey', D.number)
const RaDecKey: D.Decoder<unknown, RaDecKey> = inputKey(
'RaDecKey',
D.type({
ra: D.number,
dec: D.number
})
)
const FloatKey: D.Decoder<unknown, FloatKey> = inputKey('FloatKey', D.number)
const BooleanKey: D.Decoder<unknown, BooleanKey> = inputKey('BooleanKey', D.boolean)
const ByteKey: D.Decoder<unknown, ByteKey> = inputKey('ByteKey', D.number)
const StructKey: D.Decoder<unknown, StructKey> = D.lazy('StructKey', () => inputKey('StructKey', Input))
const Param = D.union(LongKey, RaDecKey, FloatKey, StructKey, BooleanKey, ByteKey)
const Input: D.Decoder<unknown, Input> = D.type<Input>({
paramSet: D.array(Param)
})
const input: Input = {
paramSet: [
{
LongKey: {
keyName: 'LongKey',
values: [50, 60]
}
},
{
RaDecKey: {
keyName: 'RaDecKey',
values: [
{
ra: 7.3,
dec: 12.1
}
]
}
},
{
FloatKey: {
keyName: 'FloatKey',
values: [90, 100]
}
},
{
StructKey: {
keyName: 'StructKey',
values: [
{
paramSet: [
{
BooleanKey: {
keyName: 'BooleanKey',
values: [true, false]
}
},
{
ByteKey: {
keyName: 'ByteKey',
values: [10, 20]
}
}
]
}
]
}
}
]
}
console.log(Input.decode(input))
/*
{
_tag: 'Right',
right: { paramSet: [ [Object], [Object], [Object], [Object] ] }
}
*/
This would be a possible solution if you just want to validate the input, if I understand correctly you want to decode the input to a different domain model instead
// ---------------------------
// domain
// ---------------------------
interface DomainKey<L extends string, V> {
keyName: L
values: Array<V>
}
interface DomainLongKey extends DomainKey<'LongKey', number> {}
interface DomainRaDecKey
extends DomainKey<
'LongKey',
{
ra: number
dec: number
}
> {}
interface DomainFloatKey extends DomainKey<'FloatKey', number> {}
interface DomainBooleanKey extends DomainKey<'BooleanKey', boolean> {}
interface DomainByteKey extends DomainKey<'ByteKey', number> {}
interface DomainStructKey extends DomainKey<'StructKey', DomainParams> {}
type DomainParam = DomainLongKey | DomainRaDecKey | DomainFloatKey | DomainStructKey | DomainBooleanKey | DomainByteKey // more members...
type DomainParams = Array<DomainParam>
so you can use map
// ---------------------------
// domain decoders
// ---------------------------
import { pipe } from 'fp-ts/lib/function'
const domainKey = <L extends string, V>(
literal: L,
values: D.Decoder<unknown, V>
): D.Decoder<unknown, DomainKey<L, V>> =>
pipe(
inputKey(literal, values),
D.map((a) => a[literal]) // <= map to the domain model
)
const DomainLongKey: D.Decoder<unknown, DomainLongKey> = domainKey('LongKey', D.number)
const DomainRaDecKey: D.Decoder<unknown, DomainRaDecKey> = domainKey(
'RaDecKey',
D.type({
ra: D.number,
dec: D.number
})
)
const DomainFloatKey: D.Decoder<unknown, DomainFloatKey> = domainKey('FloatKey', D.number)
const DomainBooleanKey: D.Decoder<unknown, DomainBooleanKey> = domainKey('BooleanKey', D.boolean)
const DomainByteKey: D.Decoder<unknown, DomainByteKey> = domainKey('ByteKey', D.number)
const DomainStructKey: D.Decoder<unknown, DomainStructKey> = D.lazy('StructKey', () =>
domainKey('StructKey', DomainParams)
)
const DomainParam = D.union(
DomainLongKey,
DomainRaDecKey,
DomainFloatKey,
DomainStructKey,
DomainBooleanKey,
DomainByteKey
)
const DomainParams: D.Decoder<unknown, DomainParams> = pipe(
D.type({
paramSet: D.array(DomainParam)
}),
D.map((x) => x.paramSet) // <= map to the domain model
)
console.log(DomainParams.decode(input))
/*
{
_tag: 'Right',
right: [
{ keyName: 'LongKey', values: [Array] },
{ keyName: 'RaDecKey', values: [Array] },
{ keyName: 'FloatKey', values: [Array] },
{ keyName: 'StructKey', values: [Array] }
]
}
*/
@gcanti Thank you for elaborate sample, this works well.
One question though, in case of invalid data, for example, if I pass strings in values for IntKey, it will try to decode with all the keys? To avoid multiple parsing for invalid data, I was keeping Keys -> Decoder record and based on key, I was extracting corresponding decoder to avoid decode attempt with rest of the keys as they are anyway going to fail.
In above case, I see following output:
// -------------------
// Input
// -------------------
const intParam = {
paramSet: [
{
IntKey: {
keyName: 'epoch',
values: ['1', 2, 3]
}
}
]
}
const decoded = Input.decode(intParam)
if (isLeft(decoded)) console.log(D.draw(decoded.left))
// -------------------
// Output
// -------------------
required property "paramSet"
└─ optional index 0
├─ member 0
│ └─ required property "IntKey"
│ └─ required property "values"
│ └─ optional index 0
│ └─ cannot decode "1", should be number
├─ member 1
│ └─ required property "LongKey"
│ └─ cannot decode undefined, should be Record<string, unknown>
├─ member 2
│ └─ required property "RaDecKey"
│ └─ cannot decode undefined, should be Record<string, unknown>
├─ member 3
│ └─ required property "FloatKey"
│ └─ cannot decode undefined, should be Record<string, unknown>
├─ member 4
│ └─ lazy type StructKey
│ └─ required property "StructKey"
│ └─ cannot decode undefined, should be Record<string, unknown>
├─ member 5
│ └─ required property "BooleanKey"
│ └─ cannot decode undefined, should be Record<string, unknown>
└─ member 6
└─ required property "ByteKey"
└─ cannot decode undefined, should be Record<string, unknown>
[Reference] Here is the complete implementation that we currently have with the approach I mentioned in the beginning:
https://github.com/tmtsoftware/esw-ts/blob/pritam/io-ts-decoder/lib/src/models/params/Key.ts https://github.com/tmtsoftware/esw-ts/blob/pritam/io-ts-decoder/lib/src/models/params/Parameter.ts
@kpritam you are right, DomainParam
const DomainParam = D.union(
DomainLongKey,
DomainRaDecKey,
DomainFloatKey,
DomainStructKey,
DomainBooleanKey,
DomainByteKey
)
can be optimized