Research of existing implementations by language

[Relequestual]

Problem: We do not know which languages have high quality implementations.

While members of the core team / TSC (not yet formally formed) have implementaitons in different languages, we do not know if we have "high quality" implemenations in all major languages.

Having high quality implementaitons in all major languages would improve the ecosystem. If we can identify gaps in implementations, we could then target resources to support specific languages or implementations. Overall, this would improve our ecosystem.

This item is just about the research and report, and not about making any actions or decisoions as a result of such a report.

• [ ] Decide on a list of ~10-15 programming languages or environments to assess (by some combination of popularity, TIOBE, recent publicity, commonality of use for API-adjacent or data validation-adjacent use cases, etc.)
• [ ] Compare this list against the JSON Schema implementations page
  • [ ] Confirm this page seems correct for the programming language by searching independently for JSON Schema implementations in GitHub search, Google, etc.
  • [ ] Extract a crude "health" metric of how good support for JSON Schema is in the language based on version availability, maturity of the implementation, activity of development in the implementation and compliance with the specification (time boxed to not be fully in depth)
    • does it fail some very large number of tests
    • does it support recent versions of JSON Schema
    • does it have recent releases
    • does it have recent commits
    • does it have open security warnings
    • does its maintainer consider it production-ready (or explicitly the reverse)
    • how many open issues and pull requests does it have
    • does it implement many/all/some of the interfaces from #408
  • [ ] Document languages which are popular but which are deficient in their JSON Schema support
• [ ] Update any out of date information on the implementations page on the website
• [ ] Write a blog post report for general consumptions about the method, findings, and resulting insights. (And maybe potential future work)

Assessed as high impact/low effort during our collaborators summit 2023.

[Julian]

(A brief task list is now in the issue description!)

[gregsdennis]

Have we decided on what "high quality implementation" means?

I suspect some of the results of https://github.com/json-schema-org/community/issues/408 (interfaces) will be part of that definition.

[Relequestual]

Could a work product of this include a published report? Target readership being technical leads and senior leadership.

[benjagm]

As peer the last OCWM #429: @Julian it would be great if we add action items to this plan to describe what is our understanding of high quality implementations.

[Julian]

Hey! Can you guys fill me in on what was meant by

describe what is our understanding of high quality implementations.

Are you saying something more elaborate than the task in the description? I.e.

Extract a crude "health" metric of how good support for JSON Schema is in the language based on version availability, maturity of the implementation, activity of development in the implementation and compliance with the specification (time boxed to not be fully in depth)

[Relequestual]

I think that's it, yeah. Maybe you'd think of that as a "further research" task, and so out of scope. And maybe you're right. I think we were expecting to see something less crude and more specific in terms of criteria that can be reported on a regular basis. Something we can later use to demonstrate the developing maturity of the ecosystem.

[gregsdennis]

Yeah that's pretty much it. The question came up in the last call, and scanning the issue, I didn't see anything, so I commented. Then @benjagm posted the same question as a meeting update.

[Julian]

Got it, here are a few more specifics then, which I'll also add to the description:

• does it fail some very large number of tests
• does it support recent versions of JSON Schema
• does it have recent releases
• does it have recent commits
• does it have open security warnings
• does its maintainer consider it production-ready (or explicitly the reverse)
• how many open issues and pull requests does it have

I don't know that I believe in any objective summary metric for these (i.e. one single "yes it's good" or "no it's not") -- I don't believe such a metric exists in general for software, and JSON Schema implementations being no different in my opinion -- someone using them may however use these factoids to decide whether it's good or not I think. Perhaps watching for "improvement" on that list though is a good way to measure maturity.

[Relequestual]

I think it's worth noting in any results we do publish, that things like "recent releases or commets" are not always good indicators of anything on their own. If an implementation is 100% compliant and not javascript (ha), it's probably fine and doesn't need updates.

[Julian]

100% agree -- I think the same thing is true for "how many open issues does it have", some projects have lots of feature requests and some not, I think these are the exact sort of things which make a full summary metric basically impossible/useless.

[delixfe]

For the "health" metric, the JSON-Schema-Test-Suite should play an important part. The criteria could be:

• JSON-Schema-Test-Suite runs in the release flow
• the tests are green for every supported schema version

[gregsdennis]

@delixfe that's definitely working into the equation, however we also have to recognize that an implementation may pass a majority of the tests but still have open, unaddressed issues, or it won't be updated when new tests are added, or other things.

[Relequestual]

After a recent chat, @Julian will be writing up a PR that includes a reporting template for this work.

[Julian]

I've put said initial PR up for review: https://github.com/json-schema-org/community/pull/554 with essentially the same criteria as what we were discussing. Let me know what y'all think.