Metadata for schema definition?

[vemv]

Unrelated to https://github.com/metosin/malli/issues/125

Problem statement

The current data notation for defining schemas can be considered pretty verbose:

[:map
 {:closed true} 
 [:y {:optional true} int?]]

It's not something I'd enjoy typing or reading much - feels cluttered.

Proposed solution

Accept metadata for each vector such as [:map, [:my-property etc

^:closed
[:map
 ^:optional
 [:y int?]]

Interestingly, metadata tends to get its own syntax highlighting in editors, so when reading these schemas it can feel less cluttered (since one can train the eye to fixate less on metadata).

The tradeoff is that the resulting schemas, while still data, would need expansion if consuming them for arbitrary purposes.

;; Malli would understand this piece of data OOTB
;; however other consumers would need a little helper to expand metadata into data.
;; Malli could expose such as helper as a public API.
(def MySchema ^:closed [:map ...])

Maybe Malli could offer its own def that would perform expansion OOTB.

WDYT?

Thanks - V

[ikitommi]

Would the metadata be written into properties so this would hold?:

(m/form ^:closed [:map ^:optional [:y int?]])
; => [:map {:closed true} [:y {:optional true} int?]]

the implementation would be few loc, but not sure if having more ways to do the same is better.

opening this for discussion.

[ikitommi]

example impl: #492

[rynkowsg]

I listed a few reasons why I think we should think carefully before including this feature. Feel free to ignore this comment. It is only my personal opinion.

More complex syntax, even if you don't use it, you have to document it It looks more concise for sure, still, personally I prefer the old map. The old way might be a bit more verbosive but it is easier to read, and there is less code to grasp. Less but good approaches might be easier to grasp for anyone new to the library, then a swedisch table of solutions for achiving the same result. More options is not always better.

Metadatas are not a bread and butter of a regular Clojure dev I'm using Clojure for 3 years now, and I'm not using metadata at all (besides ^:private), and from my perspective it can bring more confusion then a real good.

Bringing inconsitency with metadata usage

(meta ^:closed [:map])           ;; => {:closed true}
(meta (m/form ^:closed [:map]))  ;; => nil

It looks like we attach metadata, but we don't. I understand that we use metadata only for interpreting the data structure, but in this case, these data structures are comming back. So what we put in will no longer be the same what we take out.

It is a good idea for a utility lib This kind of feature could be implemented in external utility library. There is no reason to put it into the Malli itself. I'm not 100% sure but I think this kind of syntactic sugar data stucture transformation could be implemented outside Malli. One of the approaches could be using meander from a macro (in similar way how danzig uses meander).

Final words

I love Malli, but I'm afraid that by making it complex just by adding syntactic sugar, we wouldn't get widespread adoption. Thank you and sorry for taking your time for reading it 😅

[vemv]

Hi again, cheers for the quick POC and the healthy scrutiny.

The old way might be a bit more verbosive but it is easier to read

Easy for whom?

(there's a RH talk in which those precise words are uttered)

and there is less code to grasp.

metadata is just data, not code

I'm using Clojure for 3 years now, and I'm not using metadata at all (besides ^:private), and from my perspective it can bring more confusion then a real good.

This is an interesting point to discuss. I've used Clojure for 10 years now. Metadata indeed isn't the bread and butter of Clojure programming, however it is a tool, intentfully designed for us to use.

As a specific precedent that comes to mind RH said that ns-qualified keywords were "tragically underutilized" and the situation got gradually fixed over the following years.

Could metadata be another tragically underutilized thing?

Bringing inconsitency with metadata usage

Metadata can be preserved, nobody forces us to drop it as it is converted.

making it complex just by adding syntactic sugar

The good thing about belonging to the clj community is that we get an agreed-on meaning for "complex". Can you point out how data and metadata are inevitably complected here?

I don't think they are complected; they are composed instead. i.e. here's the data (properties), here's the metadata (directives). Different things, composed. Decomplecting is about pulling things apart.

One could go as far as saying that the current Malli syntax is a little bit complex, since data and directives get the same exact form (plain hashmaps). So they have bit of a PLOP-like positional semantic.

Similarly "syntactic sugar" is a blanket statement that can't be really proven/disproven. One man's sugar, another's tool.

[vemv]

Bringing the topic to a more practical territory, probably little would be risked/lost by having a helper in Malli core.

There isn't necessarily a need to have a one-true official syntax for everyone to use; it can be a good thing for Malli to accomodate a variety of programmers.

There's a variety of precedents of libs offering different syntaxes or options.

(the best example that comes to mind is Clojure itself; there are many ways of solving a given problem. And of formatting one's code, testing it, specing it etc.)

The alternative of maintaining a 3rd-party lib would seem somewhat prone to fall out of sync. Perhaps Malli at some point would be tempted to make a change that couldn't be retrofitted as nicely to metadata notation.

So perhaps by making it official, we would have the healthy constraint that any "data syntax" also has to be "metadatable". Perhaps a notation that wasn't metadatable would have a bigger underlying problem e.g. excessively PLOP-y.

[vemv]

Expanding on PLOPiness, for the example

[:y {:optional true} int?]

Is {:optional true} a property of :y or of int??

The answer is obvious to any Malli user, however there isn't an intrinsic reason for that semantic. It's just an arbitrary syntax that we choose to follow/remember.

By using metadata, one doesn't use a bespoke syntax at all, so it is unmistakable that in:

 ^:optional
 [:y int?]

^:optional refers to the whole data structure [:y int?], and not to something else.

So one uses an officially fostered manner to denote semantics, which might align nicely with programmers' experiences, or with other libs, etc. i.e. one reduces the cost of context switching.

[rynkowsg]

Can you point out how data and metadata are inevitably complected here?

I meant that adding additional syntax/method to do what can be done using the existing method, leads us to more code, more documentation, more design decisions, more confusion, and I think it can't make the result simpler.

Metadata can be preserved, nobody forces us to drop it as it is converted.

(m/form ^:closed [:map ^:optional [:y int?]])

What would you return for this example?

;=> option 1: [:map {:closed true} [:y {:optional true} int?]] - translate to schemas/childs properties
;=> option 2: ^:closed [:map ^:optional  [:y int?]] - if defined as metadata, return as metadata
;=> option 3: ^:closed [:map {:closed true} ^:optional  [:y {:optional true} int?]] - both, metadata and schemas/childs properties

I mean, I'm not against metadata entirely, but in the Malli schema, it looks like they complicate the overall design. They create additional decisions you need to make, and that feature doesn't bring to the overall solution that much value to pay that price.

[:y {:optional true} int?] Is {:optional true} a property of :y or of int??

I'm not 100% sure that, so please correct me if I'm wrong, but in every Malli's vector structure, properties refer to the first element in the vector. In this case {:optional true} refers to child :y. Samilary in [:y {:optional true} [:int {:max 50}] the {:max 50} referst to :int schema. I think this is a very simple and elegant design. It is positional semantic in fact, but let the first throw a stone the one who never used a list with defn followed by optional string, vector and yet another list.

^:optional refers to the whole data structure [:y int?], and not to something else.

I think this is yet another proposal, this time of inheritable properties, which is I think outside of the scope of this proposal. And this one would introduce yet another design decision to make:

^:zero [:map [:x
              [:map [:a
                     [:map [:i
                            [:map [:ii int?]]]]
                     :b int?]]
              :y int?]]

If :zero means, apply 0 as a default, how it should be translated when using s/form:

• add a new property/metadata with each node
• or as an uber-property/metadata we haven't had yet?

The alternative of maintaining a 3rd-party lib would seem somewhat prone to fall out of sync.

Yes, that is the risk. On the other hand, if it gets to the lib, it will stay and it will not be possible to get it out without breaking change. So one must make a tradeoff, one way or another.

[vemv]

I'll close the issue - nowadays my favorite way of using Malli is to build my own stuff on top of it, so I'm not as interested in any particular 'API' change.