1068 Generating context docs from schema files

[TheJuanAndOnly99]

resolves #1068

First iteration is complete:

• Created a script that generates markdown files from schema files; to execute run yarn schema2markdown in the root directory.
• The script generates context and api markdown docs from schemas
• Links are generated for each reference encountered
• Sidebars entries are automatically appended to sidebars.json
• The script iterates across all versions
• Each object definition maps to a file, as opposed to the previous version

Todo:

• confirm markdown page layout
• code cleanup

Preview - https://issue-1068--exquisite-otter-12d363.netlify.app/docs/context/schemas/Action

[linux-foundation-easycla[bot]]

The committers listed above are authorized under a signed CLA.

• :white_check_mark: login: TheJuanAndOnly99 / name: Juan Estrella (b052b86399ac8bbb55907111d41d0b86f13a4c61, 8b8960c948bf717d4c067fadc8a8cdcca9ac0f46, 69aededc8d808f6f402510a34bd6ddbac6c5e9b8, 542304105bebd5ab308c3fba352c7f7374bddffd, 814fd93d8e6bc30dce8c09ab639197f9c98637ab, 6bf17dad20aaa79863e1e51e272baee9bf8a143e, d5464a73c900bd8f2cae2e0610e4e3b03b15fec1, 1a8591139e5a224ed456a04c561f9aa95e920ca2, c449b9edc65a6d82fb7d64ed8b1e5c117e4322e4, 3d38f8333cafcdfeeb9586a30ba990e546bc78a7, 11ad90f652e943c67604fd643c0c5aed54153e01, 3ebd43a3b02217c4cd33e2b45041277987da1025, c9c2b1dfe2f96a975462c5dd544e2480cd02753c, 9eae468a6aa42c411bea6fb3c27e75ee3d2276ba, 6060696c5d3c7c2a637fc73ce71fe84921ac0824, 1950f733b472ab56133fd0f4250c46c74ff37a9f, 490af28f01a33026bf599925a8ec07715af2652d, 6dc4331a047d3876ae3978135200f5fb93e923f9, 18cee47a90ed83e91ba6175ab3693446d729f311, 1baf110ff87ce52053227087be7144ebb314290d, ce8787a9fed847127e80219a9688201773586992, 54477ae4f38c9ad5a12a6131c0b3f4050273bb4b, eed96027d2863cff1d3ba2dc53e787090bf9e42a, 44e0c66d113658547f8d6df49adc7c357984c893, 0e0c72b2e6f8ecf98497c1a2e89ea59bd2623912, fee515f61b62a0203b243f93d2328ca651ec86b2, f4a80aac3e31d0415f729b87a6545f793d92c164, 47df2e3751b87d011a5f1cdd6efcf24e15a98ab3, 901d72a37969427868e2e5fc42df30f94a44f850, eac8599a5484d187161fc499d97503e473c0274b, a2b12f7c073aab47cbcc570d210dbfbe829d7954, b27c3f2ce9e4d7e492538584b5ca2048172ae560, 2f8bd006d0ddac1f0da7c69fb1e8b1c01c428246, 86d994aed1ff4fa3afe698d62003be645ebc2b55, 0c83f6aae1458d7550da8c3b20dcc6049b7cadf9, fe2e40e793fe8abb9e428f78bcd9ddb5787ed1e4, 4c49bb5dfb39ba45e5d453354d9100170dd98400, c51fe3fe4a97b18d4bbbb89ae18f3f1a361cb802, 816c3a81fc039a21b9d38c507f8d16e167229f5d, 9cb323a6f6c08c83e433ddeb131851ad4528c27d, a0216b68390aeb470237a8698ca5bfcc158f0ffb, 0b13c7aeda867875fe2b0c57bf62e797a818c039)
• :white_check_mark: login: maoo / name: Maurizio Pillitu (a956cecc54decd8f9ffc913702edca46ee52b9ab, 5b915073e0a80926dd811204b68ac3fe14a22805, 3a766959f9332d3e1b62677c6e048d9836f81e8b, f74e8b0e59a0c269e0e2db9f2cfdaf02b07fba77, 297539ca7e2f295744ccfa133ad684d112c32b16, f90bdceb482273f7c5fc392aea7adeea0ce8790a, b8ab9c5a6a1cfef52db8bde0669bfd350979f922)

[netlify[bot]]

✅ Deploy Preview for fdc3 canceled.

Name	Link
🔨 Latest commit	eed96027d2863cff1d3ba2dc53e787090bf9e42a
🔍 Latest deploy log	https://app.netlify.com/sites/fdc3/deploys/66339c896bdbf80008eb24bc

[TheJuanAndOnly99]

@kriswest @robmoffat - first iteration is done; please check the description above to see what's been done and what has not; there's also a preview on Netlify pointing to my fork (the build command will need to be updated when we go live with this).

Eager to hear your feedback! /cc @mistryvinay

[maoo]

Hi @julianna-ciq , thanks SO much for the detailed review! @TheJuanAndOnly99 is currently off, he'll be back on March 3rd and work on your comments.

[TheJuanAndOnly99]

Hey @julianna-ciq thank you so much, great review, really appreciate your feedback. I've implemented all your notes. Would you mind reviewing again. If everything looks good do you think it's okay to merge?

(Please note that after this is merged we would have to update the build command on Netlify).

[TheJuanAndOnly99]

Thanks again @julianna-ciq! I've implemented both suggestions.

@robmoffat added a "Generated from" line at the bottom that points to the Json schema file.

Can you give me a clear mapping between the schema and the typescript (and c#) files to point to?

[kriswest]

Can you give me a clear mapping between the schema and the typescript (and c#) files to point to?

@TheJuanAndOnly99 The mapping from schema to typescript is easy because this script: https://github.com/finos/FDC3/blob/3c2af8b33c203f46a2b4cd6e5de48fa558a78538/package.json#L28 will generate this file: https://github.com/finos/FDC3/blob/main/src/context/ContextTypes.ts

which contains typescript for every schema in the /schemas/context folder.

However, that doesn't give you a simple link to use from a particular docs page to the TS, but it does mean we can add some advice somewhere about where to get TypeScript types from - which I am surprised to report that we are not actually doing anywhere! It should probably be added as a section in the context overview: https://fdc3.finos.org/docs/context/spec

[maoo]

@kriswest @robmoffat @TheJuanAndOnly99 - what if we tackle the Typescript mapping as part of a new PR ? I think there's already enough to merge/review here, and I'm worried about adding more to this PR scope.

This also allows someone else from the community to step in and volunteer.

[robmoffat]

makes sense to me @maoo

[kriswest]

@TheJuanAndOnly99 merging changes from upstream:master/main and re-running npm install will probably resolve the issues with the checks you're having above. This should sort out your fork:

git fetch
git branch --move master main
git push -u origin main

Then update main from upstream:main and then merge into your branch.

[kriswest]

@TheJuanAndOnly99 one more issue we spotted in an important schema (instrument):

• New: https://issue-1068--exquisite-otter-12d363.netlify.app/docs/context/schemas/Instrument
• Old: https://fdc3.finos.org/docs/next/context/ref/Instrument

First, ignore the difference in the description - I moved the notes into the id property in the schema as they all related to that. However, what we've lost is the details of the standardized subproperties of id and of market. Its going to need to pick those up and show them in the generated page...

Hence, if a property is an object or array, you probably need to descend into it and handles its own properties, unless its a ref to another defined type...

[maoo]

@TheJuanAndOnly99 one more issue we spotted in an important schema (instrument):

• New: https://issue-1068--exquisite-otter-12d363.netlify.app/docs/context/schemas/Instrument
• Old: https://fdc3.finos.org/docs/next/context/ref/Instrument

First, ignore the difference in the description - I moved the notes into the id property in the schema as they all related to that. However, what we've lost is the details of the standardized subproperties of id and of market. Its going to need to pick those up and show them in the generated page...

Hence, if a property is an object or array, you probably need to descend into it and handles its own properties, unless its a ref to another defined type...

@kriswest we made the changes, could you please review https://issue-1068--exquisite-otter-12d363.netlify.app/docs/context/schemas/instrument/ ?

[maoo]

@TheJuanAndOnly99 merging changes from upstream:master/main and re-running npm install will probably resolve the issues with the checks you're having above. This should sort out your fork:

git fetch
git branch --move master main
git push -u origin main

Then update main from upstream:main and then merge into your branch.

@kriswest - we did that, but since then all my commits fail with /Users/m/w/FDC3/src/internal/typeHelpers.ts - Parsing error: Cannot read properties of undefined (reading 'map') ; was there any change to this file recently? Any tips on how to address this?

[kriswest]

@kriswest - we did that, but since then all my commits fail with /Users/m/w/FDC3/src/internal/typeHelpers.ts - Parsing error: Cannot read properties of undefined (reading 'map') ; was there any change to this file recently? Any tips on how to address this?

Yes, a PR got merged today that added that. I'll see if I can replicate quickly.

[kriswest]

I can replicate. Its from the linter, which is setup to lint src on commit, where the lint command only does src/api. Not exactly sure what its problem is however...

[kriswest]

@maoo see https://github.com/finos/FDC3/pull/1187 for a solution (needs review and merge)

[kriswest]

...and you might need to delete package-lock.json

[kriswest]

Is this ready for re-review @maoo and @TheJuanAndOnly99 ? Its still marked draft so just checking. If its ready I can encourage the @finos/fdc3-maintainers and @finos/fdc3-editors to review and merge

[kriswest]

@TheJuanAndOnly99 schema2markdown is tripping one of the semgrep rules: https://github.com/finos/FDC3/actions/runs/8837630351/job/24266920820?pr=1151

and the build failed, probably indicating that you need to run npm install in your branch (possibly also delete the local file first): https://github.com/finos/FDC3/actions/runs/8837630352/job/24266920775?pr=1151

[TheJuanAndOnly99]

Hi @kriswest I fixed the build issue but I don't see it triggering the check again.

[kriswest]

@TheJuanAndOnly99 I've manually re-run it. I know its going to fail (as the docusaurus website has developed an issue with a dependency, but the log will tell us whether anything else is still amis).

[kriswest]

@TheJuanAndOnly99 you're all clear! Just he axios dependency issue in the log

[kriswest]

@mistryvinay and I should be able to take it from here - we need to run through another review and then make the tweaks necessary to have the generated pages replace the existing ones, meaning they'll need to generate onto the same URLs as the manual pages, which we'll delete!

[kriswest]

...although @TheJuanAndOnly99 you might need to explain or handle whatever is needed regarding the netlify preview/build... I don't yet know what change is needed nor why.

If its about making sure the generation script gets run, we could possibly do that with a commit hook and npm package script...

[TheJuanAndOnly99]

@kriswest when this PR gets merged I need to update the build command on Netlify to run the script and that will take care of things for us.

[kriswest]

@TheJuanAndOnly99 The preview for this appears to be broken - could you take a look, not seeing the generated pages anymore. Perhaps a conflict on updating from @robmoffat's homepage update

[TheJuanAndOnly99]

Hi @kriswest the preview is running on my personal Netlify (since the build command is different). You can find it here.

[kriswest]

@TheJuanAndOnly99 I'm not seeing the generated pages in the preview's sidebar anymore:

These are the old pages?, right: https://exquisite-otter-12d363.netlify.app/docs/next/context/ref/Action

[TheJuanAndOnly99]

@kriswest yep you are correct! I think I've fixed it. https://issue-1068--exquisite-otter-12d363.netlify.app/docs/context/schemas/Action

[kriswest]

The missing id and name properties should all be fixed by : https://github.com/finos/FDC3/pull/1233