FDC3 Replace existing Context pages with JSON schema definitions

Enhancement Request

Use Case:

Generate Docusaurus pages for Context types using JSON schema files.

Proposal already created for markdown docs, which can be found here #1020 , which is included as part of release 2.1

Work required

Either:

Script to generate static markdown documentation from JSON schema files
Build or integrate a JSON Schema viewer solution

and:

Generate the Docusaurus front matter, programmatically modify the sidebars file to make sure that they're linked and they're in the website or the viewer.

Additional Information

Discussion around this particular enhancement is being tracked via #940 and #1059

Sep 20 '23 12:09 mistryvinay

Thanks for raising this @mistryvinay, I made a slight tweak to the description (docs are in 2.1 release and we could either build a viewer or generate static markdown)

Sep 20 '23 13:09 kriswest

JSON Schema documentation rendering notes from @kriswest
- JSON Schema -> Markdown or HTML conversion:
  - https://tomcollins.github.io/json-schema-static-docs/
    - Node module
    - Generates static markdown docs
    - Example:
      - HTML: https://tomcollins.github.io/json-schema-static-docs/examples/person.html
      - Generated markdown source: https://raw.githubusercontent.com/tomcollins/json-schema-static-docs/master/gh-pages/examples/person.md
  - https://pypi.org/project/json-schema-for-humans/
    - Python module
    - Docs: https://coveooss.github.io/json-schema-for-humans/#/
    - Only supports up to JSON Schema Draft-07
    - Can generate JS/HTML or Markdown, number of template options
    - Examples: https://coveooss.github.io/json-schema-for-humans/#/Examples
  - https://github.com/CesiumGS/wetzel
    - https://github.com/CesiumGS/wetzel#example
    - Node module
    - Generates static markdown docs
    - Example: https://github.com/CesiumGS/wetzel#example
  - https://github.com/interagent/prmd
    - Ruby
    - Generates static markdown docs
  - https://github.com/BrianWendt/json-schema-md-doc
    - Node module
    - Generates static markdown docs
    - Example:
      - Live: https://brianwendt.github.io/json-schema-md-doc
  - https://github.com/adobe/jsonschema2md
  - https://www.npmjs.com/package/json-schema-to-markdown
  - https://npmjs.com/package/json-schema-to-markdown-table
- JSON Schema viewers (have to be embedded on a static page)
  - https://github.com/lbovet/docson
    - Example: http://lbovet.github.io/docson/index.html#/docson/examples/example.json
  - https://github.com/yanick/jsonschematic/
    - Example: https://yanick.github.io/jsonschematic/
- Docusaurus plugins
  - https://jy95.github.io/docusaurus-json-schema-plugin/
    - Only supports up to JSON Schema Draft-07
    - Viewer AND editor
    - Example:
      - https://jy95.github.io/docusaurus-json-schema-plugin/boxFileSchema
      - https://jy95.github.io/docusaurus-json-schema-plugin/docs/demo/viewer

Nov 15 '23 13:11 kriswest

@TheJuanAndOnly99

Jan 15 '24 15:01 robmoffat

So far the Docusaurus schema browser is displaying some but not all of the JSON data for me.

I'm also looking at the Tom Collins repo and will update this post tonight.

EDIT 1:

I have been using the Docusaurus Playground to see how the action schema renders: Playground | Docusaurus

The Docusaurus playground shows unsolved refs from the FDC3 Context schema. I have never really used JSON for a project before and am looking into the $ref structure.

This user schema from Tom Collins renders without any issues in the Docusaurus Playground: Draft 07 - User Example

I think using JSON Schema Static generator from Tom Collins to get markdown that renders in Docusaurus will be successful. I will update this post with screenshots.

Jan 17 '24 14:01 fapulito

that's going better for you than it went for me, then. I couldn't get that Docusaurus plugin going at all

Jan 17 '24 18:01 robmoffat

fat fingers

Jan 17 '24 18:01 kriswest

Some to try out:

https://github.com/finos/FDC3/blob/master/schemas/context/chart.schema.json
https://github.com/finos/FDC3/blob/master/schemas/context/chatInitSettings.schema.json
https://github.com/finos/FDC3/blob/master/schemas/context/instrument.schema.json

Jan 19 '24 13:01 robmoffat

Discussed whether it might be better to have a docusaurus plugin, or use a node script.

Jan 19 '24 14:01 robmoffat

I'd suggest to experiment a bit with https://github.com/adobe/jsonschema2md (or similar tools).

WDYT? Did anyone had experience with this tool?

If we all agree, I'd suggest to proceed with:

run the schema-to-markdown transformation of the files mentioned by @robmoffat 2 comments ago
share the result on this issue and decide to move forward or not
embed the generated markdown into Docusaurus (here there are several approaches, we can talk about this when we get there)
integrate MD generation into Github actions and Netlify publishing

@robmoffat @kriswest @mistryvinay - Thoughts?

Feb 01 '24 11:02 maoo

Here's an example output from jsonschema2md: Schema: https://github.com/adobe/jsonschema2md/blob/main/examples/schemas/complex.schema.json Markdown: https://github.com/adobe/jsonschema2md/blob/main/examples/docs/complex.md

The problem is often referencing (in all the different rendering tools), which can result in somewhat difficult to read output or incorporating the entire documentation for the referenced type. Our own experience with generating documentation from this was poor TBH and we ended up rolling our own custom solution. It's not terribly hard to do as the schemas can be imported as JSON objects and processed. The advantage is that it allows you to bake in some knowledge of the how your schemas (and their docs) are structured into the rendering. This results in something inflexible, but better fitted to use case - I briefly ran through the code we use to this and can dig out some excerpts from it if useful.

Anyway that's my 2 cents - I worry that adapting a general-purpose tool will take us longer than coming up with custom rendering code (to either HTML or Markdown).

Feb 01 '24 11:02 kriswest

Thanks for the feedback @kriswest - very helpful.

Let's go with the custom solution; I guess that the biggest task here is to define some requirements, for example:

Page layout - a Google Drawing - or similar - would be great; something visual I'd say
Content mapping - based on the page layout, specify which content blocks are coming from which part of the schema Let me know if there's anything else that comes to mind.

Re. the implementation, I see 2 different directions, and I'd like to hear your opinion:

Generating Markdown files at build time (Netlify) from JSON schema and copy them into the docs/ folder
Transforming JSON Schema to Markdown at runtime, embedded into the logic of a specific (mdx) page; we may have to deal with page routing here

Eager to hear your thoughts!

Feb 01 '24 13:02 maoo

@maoo when we met with Juan and Rob on this I suggested trying to reproduce the existing layouts/markdown, e.g.

Current page: https://fdc3.finos.org/docs/context/ref/Chart
markdown source for it: ttps://raw.githubusercontent.com/finos/FDC3/master/docs/context/ref/Chart.md
Schema file: https://github.com/finos/FDC3/blob/master/schemas/context/chart.schema.json

You'll note I've replicated all the content from the markdown doc in the schema file for every standardized context already. The only exceptions are:

the 'see also' content, which we can either drop or replace with links to the page for any referenced context.
we are not tied to the table, we just need equivalent content in whatever form is convenient.

Re: mapping, the top level title field gives the page title, the description field the initial description:

https://github.com/finos/FDC3/blob/f4a46c0bf64cb72abf1343ad34ac2e6f251b000c/schemas/context/chart.schema.json#L5-L6

The same pattern applies to the properties:

https://github.com/finos/FDC3/blob/f4a46c0bf64cb72abf1343ad34ac2e6f251b000c/schemas/context/chart.schema.json#L9C7-L13C17

You do need to descend down the properties object in case there are multiple levels, which is rare, its usually a reference to another type (we don't need to include all the docs for the references just link to their own page).

Finally there is an example element in each schema that provides the code block:

https://github.com/finos/FDC3/blob/f4a46c0bf64cb72abf1343ad34ac2e6f251b000c/schemas/context/chart.schema.json#L45C3-L87C4

Had I a spare afternoon (ever), I'd have a go at this as it seems like a fun little script to write ;-)

Feb 01 '24 17:02 kriswest

@maoo Re:

Re. the implementation, I see 2 different directions, and I'd like to hear your opinion:

Generating Markdown files at build time (Netlify) from JSON schema and copy them into the docs/ folder Transforming JSON Schema to Markdown at runtime, embedded into the logic of a specific page; we may have to deal with page routing here

I proposed these same two options. I would probably start with the former and write a function that generates markdown by parsing the schema and then investigate whether you can improve on it and do it at runtime via react, see: https://docusaurus.io/docs/markdown-features/react. The parsing is going to remain the same, rather its the packaging of it and the output generation (which would be html/react in this latter version). I did wonder about the page routing - you could have one page for all content docs (with anchors for specific ones)... the real problem is the navigation which is currently manually maintained (in this file): https://github.com/finos/FDC3/blob/master/website/sidebars.json That nav will either have to be pregenerated (so first approach with a build script) or moved within the page - but in a way that preserves the ability to link direct to the page...

Hence, for the sake of a simple life, I'd start with generation at build time - but that's just how I would approach it :-).

Feb 01 '24 17:02 kriswest

Hence, for the sake of a simple life, I'd start with generation at build time - but that's just how I would approach it :-).

That's my personal preference too, so we're fully aligned; and as you said, we can still reuse the logic and adopt the runtime approach later on.

Thanks for the recap, I think that we have everything we need, I'll let @TheJuanAndOnly99 take the lead on this and start playing around with code.

Feb 01 '24 19:02 maoo