SigMF
SigMF copied to clipboard
sigmf
SigMF Core JSON Schema and new method of creating PDF
What I've added is a first cut of a system that generates a PDF using latex, I didn't want to add the pdf for sake of git space so you can find it here - https://raw.githubusercontent.com/777arc/777arc.github.io/2e66ae0b6ea462cdad6f17c213f929797a7a88e6/sigmf-spec.pdf
and just an aside- github renders pdfs in-browser decently well, although you cant use the links or copy text- https://github.com/777arc/777arc.github.io/blob/master/sigmf-spec.pdf
A separate md file contains everything in the current .md file that isnt in the JSON schema. So the idea would be we remove sigmf-spec.md and switch to pointing people to the pdf, and the source content would all come from the schemas, pretables.md, and posttables.md.
SigMF_Objects.png is temporary, until a nicer diagram in draw.io is made, added as a xyz.drawio.png file.
If anyone likes playing with LaTeX feel free to fix little things you find.
Is this schema a re-write of the schema-meta.json from the sigmf-python module or an attempt from scratch?
It seems very similar with some updates.
The schema itself looks fine, but a few questions:
- In the long term should
sigmf-pythonmirror this schema or retrieve it somehow? - Will there be a schema for the collection file (like this)?
- Should the version of this schema be more apparent? I see it in the
core:versionas a default. Is that were libraries should be detecting this schema version?
It's not a re-write from scratch because I think they both originated from the same source, but the idea is for this one to be the schema, in fact we even talked about having our main sigmf markdown specs generated from the json schema, so that the schema is the source of truth, but that can be a future discussion. So yeah we'll need a way to mirror or reference this one from within the python module. It's not unusual for json schemas to be referenced via a URL, not sure if that would work for the python module.
How about this approach-
- We first finalize this schema and get it merged in (I'll incorporate what I missed from the python module one incl versioning at the top level)
- We'll figure out the cleanest way for the python module to switch over to this one
- We can look into generating all or part of the markdown file from the schema, so that info like descriptions of fields only lives in one spot.
As far as Collections schema, we do need that as well, I'll either make a new file or find some way to have them both in the same schema file.
Oh it looks like the Collection schema was part of the main schema but then you had to split them into 2 to fix a test. From a validators perspective its a PITA when they are combined, so yeah we can keep them separate, even though it makes versioning a little annoying
I renamed the file, good suggestion! That PR #292 I'm going to abandon, in favor of completing this PR, then doing a follow-up PR where we generate all or part of the markdown file using the JSON schema, still some discussion needed around it, but Jacob and I really like the idea of the descriptions for specific fields only living in one spot, and what better place than something machine readable =)
Do we have a process for extension schemas to be written?
Writing the schemas is pretty painful - I've tried a few methods but doing it manually seems to be best.
If you meant "do we have a process for extensions/schemas to be merged", I don't think so. We have rejected multiple PRs for extensions in the past. If we do allow extensions into this main repo, perhaps they should have a schema to go along with them? Should all extensions be in a different repo?
Another issue is testing, which is why there is a schema in the sigmf-python repo. I'm still wondering how exactly i'll pull the schema from this repo instead. I considered adding this repo as a submodule, but I kinda hate those...
@jacobagilbert suggested I just use 1 big PR instead of splitting it. So what I've just added is a first cut of a system that generates a PDF using latex, I didn't want to add the pdf for sake of git space so you can find it here - https://raw.githubusercontent.com/777arc/777arc.github.io/2e66ae0b6ea462cdad6f17c213f929797a7a88e6/sigmf-spec.pdf
and just an aside- github renders pdfs in-browser decently well, although you cant use the links or copy text- https://github.com/777arc/777arc.github.io/blob/master/sigmf-spec.pdf
The pretables.md and posttables.md (names TBD) contains everything in the current .md file that isnt in the JSON schema. So the idea would be we remove sigmf-spec.md and switch to pointing people to the pdf, and the source content would all come from the schemas, pretables.md, and posttables.md.
SigMF_Objects.png is temporary, until a nicer diagram in draw.io is made, added as a xyz.drawio.png file.
If anyone likes playing with LaTeX feel free to fix little things you find.
The overall goal here is for our spec to not live in duplicate spots, while also making sure the JSON schema always matches the specs, so the idea was to have the tables/descriptions of global/captures/annotations to live in the JSON schema and no where else. The pre/post are for all the other content that was in the .md. They could be merged into 1 file if we stick some identifier python can parse in the middle to point out where the split is.
So this generated PDF would entirely replace the sigmf-spec.md, in fact I should have just removed sigmf-spec.md in this PR to make that more clear. The idea is folks wouldnt be referencing pre/post files, they would reference the PDF as the official spec document.
This repo has always been about the core specs and extensions right, so I don't see how it expands the scope, but I agree the extra files are a little bloaty so it might be better to organize, eg have an images directory in case we add more images in the spec.
Any changes to the script or .md or json are either to 1) make an actual change to SigMF 2) clarify something 3) make the pdf look nicer, fix little issues, etc. So mainly it's a game of refining a document over time.
I'll keep going through and fixing up the little issues
I still need to get the changelog added, its important we keep a changelog of API changes moving foward
Alright I've addressed comments and gone through the commits to main since starting this effort and copied the applicable changes into the new docs/schema. Latest PDF can be found here- https://raw.githubusercontent.com/777arc/777arc.github.io/master/sigmf-spec.pdf
I also replaced the ASCII diagram with one that I think new folks will be able to understand quicker/easier
As per discussion on the monthly SigMF call today, I'm going to get the script to produce HTML version of the specs and a github action to do it, that way we can always have the HTML updated even if the pdf isnt always, since that would bloat the repo size
Alright I have the github pipeline ready, but it cant publish to github pages unless its merged into main, but it seems to be producing the right HTML and PDF