Support for user defined types

[mikedanese]

trafficstars

It would be nice to add some support for stronger typing of objects. Ideally this would manifest as runtime validation of visible fields after objects are fully reified. You could probably get pretty far implementing this functionality in a library with assert. Another alternative is to write a library that does JSON Schema validation, but the definition language is rather verbose.

[sbarzowski]

I was thinking about some way to describe objects for the purpose of documentation and if we had that, I guess runtime checks using the same description would be quite natural. If you have any concrete ideas of how it could work, please share :-)

It's not that much related, but also FYI I'm currently working on a linter which tries to infer "types" (richer than normal jsonnet types) of expressions. The main benefit will be statically catching typos and trivial mistakes (calling a nonexistent function from a library etc.). This doesn't work very well with function arguments - we know pretty much nothing about them (unless we derive it from the usage).

[mikedanese]

GCL types aren't very elegant to define and I don't think Nix supports user defined types. I saw how far I could get with a library, but some language support would really clean the syntax up.

https://gist.github.com/mikedanese/f4a4cb57fdd331acf89433575a71187a

Strawman syntax could be something like:

local mytypes = {
  type foo: {
    a: number,
  },
  type bar: {
    foo: mytypes.foo,
    foos: mytypes.foo[],
    foom: map[]mytypes.foo,
    b: string,
  },
}

Then for declaration

{
  mytypes.bar barobj: {
    foo: {
      a: 1,
    },
    foos: [
      {
        a: 1,
      },
    ],
    foom: {
      ka: {
        a: 1,
      },
    },
    b: 'b',
  },
}

[mikedanese]

I came by this config language the other day:

https://cue.googlesource.com/cue/+/HEAD/doc/tutorial/basics/Readme.md

I found the type system to be pretty interesting. It looks like each layer of config can further constrain an abstract object until all properties are concrete, or a type mismatch results in an error.

[sparkprime]

I think it's by the original author of BCL :)

[sparkprime]

I think it might be nice to have jsdoc closure style comments and then enforce them with the linter. Alternatively, some kind of types syntax extension like pytype.

[Globegitter]

I think a type syntax extension would make more sense, I think it feels more natural to write rather than attaching documentation to code.

[okulam]

Is there anyone working on type system in Jsonnet? One other alternative would be implementing the schema validation in linter, we can basically pass JSON Schema path as an argument and it can throw the relevant errors.

[sbarzowski]

I'm aware of some exploratory research, which hasn't been made public yet. I have also thought about it for quite a while and have a general idea how to fit a type system into Jsonnet. I'll be happy to discuss the possibilities is someone is interested in picking this up.

One other alternative would be implementing the schema validation in linter, we can basically pass JSON Schema path as an argument and it can throw the relevant errors.

This could be useful and maybe a very good , but it's only going to help with validating the result, I think. A full type system would allow us to check much more.

[buremba]

There is also a relevant research published on Google Groups. I think the JSDoc types is a good way to start. Ideally, we can reference JSON Schemas there and the linter automatically validates the data. If that works well, we can introduce special syntax similar to Cue as follows:

*type {
  prop: 1
}

[morlay]

Without new syntax, we could start to do validation just with a hidden field __schema__.

https://github.com/jsonnetmod/types

local t = import 'github.com/jsonnetmod/types/t.libsonnet';

local Meta = {
  __schema__:: {
    type: 'object',
    properties: {
      name: { type: 'string' },
      age: { type: 'string' },
    },
    required: ['name']
  },
};

// or with build helper
local Meta = t.objectOf({
  name: t.string(),
  age:: t.number(), // hidden means not required
});

t.validate(Meta {
  name: 1,  // should throw error like `$.name` should be a string value, but got number value
  age: 18,
})

more examples: https://github.com/jsonnetmod/types/tree/master/tests/fixtures

[sbarzowski]

@morlay Yes, I think this is a promising direction. It allows us to start describing types without any immediate change to the language.

Once we have a more concrete schema description we can add the language extensions necessary for static checking. I've been thinking quite a lot about this. If you or someone else would be interested in bringing this to the next level, please let me know – I'll be very happy to discuss it in detail!

[morlay]

JSONNET is widely used in k8s community.
Recently, I'm moving to JSONNET from Helm YAML templates. JSONNET is very cool data templates, it save me from Helm YAML templates. However, without type validation, I have to apply results to k8s for validation when i define resources.

So I really hope JSONNET could throw validation errors when compiling to json data.

Instead of validate final json data with json schema, which need i define a complex json schema, I start this project https://github.com/jsonnetmod/types to just add hidden field __schema__ with JSON SCHEMA to to validate the objects I care about.

I will create The method validate(value, schema={}, path='$') could be a wrapper when flag --validate enabled. After traverse whole JSONNET root, the invalid field should throw validation errors with the key path (even hidden fields if field __schema__ exists).

custom-jsonnet-cli --validate ./app.jsonnet

and the files could write like

// core/v1.libsonnet
// should generate from json schema of k8s
local t = import 'github.com/jsonnetmod/types/t.libsonnet';

local ObjectMeta = t.objectOf({
  name: t.string(),
  Namespace:: t.string(),
  labels:: t.mapOf(t.string()),
  annotations:: t.mapOf(t.string()),
});

local NamespaceSpec = t.objectOf({
  finalizers:: t.arrayOf(t.string()),
});

local Namespace = t.objectOf({
  apiVersion: t.const('v1'),
  kind: t.const('Namespace'),
  metadata: ObjectMeta,
  spec: NamespaceSpec,
});

// or just import the json schema direct
local Namespace = {
  __schema__:: (import 'path/to/json-schema/core.v1.Namespace.json'),
  // could defined default values to easier usage.
  apiVersion: 'v1',
  kind: 'Namespace',
}

{
  ObjectMeta:: ObjectMeta,
  Namespace:: Namespace,
  NamespaceSpec:: NamespaceSpec,
}

// app.jsonnet 
local v1 = import 'core/v1.libsonnet';

{
  values:: {
    namespace: 'default',
  },
  namespace: v1.Namespace {
    metadata+: {
      name: $.values.default,
    },
  },
}

@sbarzowski If you and JSONNET community accept this way, we could make it as built-in. But one important question here:

Which version of JSON SCHEMA we used ?

the newest 2020-12 have breaking changes from previous versions. and the widely-used OPENAPI using a custom version of JSON SCHEMA (https://swagger.io/specification/#definitions)

I don't think we need to create another json schema. may just choose one.

---

BTW, I'm a little worry about, bringing type system to JSONNET is not a good idea. We could see the changes of TypeScript, which extends JavaScript by adding types, features added, more and more syntax added, still not enough to validate full data struct.

[brettviren]

+1 @morlay

I would be very interested to have a way to apply a schema at the point at which a schema-described value is constructed in the Jsonnet. IOW, a way for the developer of a Jsonnet data structure to know exactly where a schema violation was introduced in the Jsonnet code (not having to wait until after the full structure is built).

I've been developing something along these lines (called "moo"). It lacks an in-Jsonnet validator which would be very cool to have. Instead, a moo schema is expressed in Jsonnet and can be transformed to JSON Schema to validate some objects, post-construction. The tooling for that validation is in Python. The goals of "moo" also include code generation and schema-driven object construction (giving the early-error, valid-by-construction pattern described above but again in Python, not Jsonnet) which I think are further afield than this issue. But, maybe worth looking at nonetheless.

https://brettviren.github.io/moo/

Trivial moo schema:

https://github.com/brettviren/moo/blob/master/examples/oschema/sys.jsonnet

A more complex moo schema that also depends on the trivial one:

https://github.com/brettviren/moo/blob/master/examples/oschema/app.jsonnet

And, here are the schema constructor functions

https://github.com/brettviren/moo/blob/master/moo/jsonnet-code/oschema.jsonnet#L111

[sbarzowski]

Cool stuff! Thank you both for exploring this area!

If you and JSONNET community accept this way, we could make it as built-in.

Supporting the official JSON Schema validation would be really great. I would keep it as a library for now. If we see some performance problems, we can perhaps create builtin helpers.

Which version of JSON SCHEMA we used ?

My intuition is to go with the newest official version. I don't have much experience with it though.

I don't think we need to create another json schema. may just choose one.

I think the ideal solution for type system and schema will require something custom. Some reasons:

1. Jsonnet has more values than JSON, e.g. we also have functions and distinction between visible/hidden fields.
2. JSON schema is pretty complicated. We might not be able to support the whole thing from day 1. At the same time a lot of more complex things in it could just be expressed as Jsonnet functions.
3. I would like a more formal specification for our official schema/types. It's hard with JSON Schema since it's a 3rd party complex standard.
4. JSON Schema has its own "import" mechanism with references to remote urls. This is a big no-no in Jsonnet. It's possible to work around it by passing a mapping from url to content explicitly (it doesn't even need to pollute the code much – it can be done at a library wrapper level).
5. There already exist multiple incompatible dialects of JSON Schema.

I think this way we could have something which fits better with the rest of the language.

That said, even if we had that, it would be really cool to also have support for JSON Schema validation (since it's a common standard).

BTW, I'm a little worry about, bringing type system to JSONNET is not a good idea. We could see the changes of TypeScript, which extends JavaScript by adding types, features added, more and more syntax added, still not enough to validate full data struct.

Well, first of all, TypeScript is a huge success. The type system doesn't solve every possible problem, but still brings a lot of value.

[...] still not enough to validate full data struct

A static type system is not going to be a replacement for runtime validation. Proving some properties statically is just pretty damn hard :-). E.g. it's pretty hard to prove that a string output always matches a regexp.

At the same time a static type system has some advantages over runtime validation. It offers a faster feedback loop, allows checking code which is currently dead (e.g. because it has just been written and is not used anywhere yet). It's also better for expressing properties about functions (which are black boxes at runtime).

So ideally we would have both static and runtime validation.

[morlay]

@sbarzowski

https://github.com/jsonnetmod/types

i already passed JSON schema test cases of draft7 & draft6 😂, other versions could convert to bump.

totally agree with drop ref feature.

pattern validation need std.regexMatch, i found pr for this.

hidden field validation i have no idea to handle.

Libs & testing under https://github.com/jsonnetmod/types/blob/master/lib/

[ghostsquad]

Has any further progress been made along these lines?