Remove the SaintCoinach Schema

[slotthhy]

Functionality

• Adds support for the EXDSchema definition format
• Adds definition update support to both SaintCoinach.Cmd and Godbert
  • This update functionality will check for an updated gamever, but also for a changed schema set of the current gamever, so definitions can always be current

Removals

• Removes all json definition files
  • A lot of code that is used for definition parsing has been adjusted, so the json definitions will likely no longer work
• Removes definition update functionality
• Removes the SaintCoinach.History.zip "store" file

Fixes

• The sheet grid pane will no longer stop updating after toggling any of the offset options
  • Due to the fix for this, "Sort by Offsets" and "Show Offsets" will no longer take effect until the user chooses another sheet

Known Issues

• Some SC API-based sheet access may be incorrect due to inconsistent column names. This can be fixed, but SC is close to being retired

[Supamiu]

Hi, amazing job you've done here.

I see you mention retiring SC, I feel like we should probably keep at least the base exports abd godbert

[CrimsonOrion]

Why are we retiring SC? Is there a newer application on the horizon?

[slotthhy]

Hi, amazing job you've done here.

I see you mention retiring SC, I feel like we should probably keep at least the base exports abd godbert

Why are we retiring SC? Is there a newer application on the horizon?

Sorry, I think I was referring to retiring SC as a library. There is almost no use for SC as a library outside of applications that already use it.

[mattantonelli]

Sorry, I think I was referring to retiring SC as a library. There is almost no use for SC as a library outside of applications that already use it.

Garland Tools is the application I would be worried about. (@ClayLivince)

---

It would be good to document any breaking changes this introduces to the data. For example, it looks like brackets and braces were removed from the column names.

Other than that, EXDSchema looks very clean. I'm curious how the column ordering works, because it doesn't seem to match up with what SC currently produces.

[ClayLivince]

I am also interested with how does EXDSchema 's yaml "definition" updates.(Not very clear after reading it's code) If it is still via some manual operations, what's it's difference from current SaintCoinach definition other than formats? Just separating Definition repo and lib/ Sc.cmd/ Gobbert repo and then let them fetch the json definition does the same thing isn't it?

[Supamiu]

My understanding (and I'll let OP add details or invalidate some of it) is that:

• Saint Schema is way too complex and unadapted, because ComplexLink, Multiref etc are just making things more complicated, not to mention grouping.
• Grouping, in fact, is way easier with yaml definitions, because while SaintC schema column index order while, EXDSchema uses columns offset order (basically reading in the rows's column order, not the order defined by the header), which makes things much easier for grouping into sub objects, let's take Quest sheet as example:

SaintC defines Script into two arrays: Script{Arg} and Script{Instruction}, grouping is pretty much impossible here so we end up with two arrays, having to combine each Arg with its Instructions.

In godbert, if you enable "Show offset" in settings, you'll have Something like this (horizontal scrolling finally shows all script args then all instructions):

If you also enable "Sort by offset", you'll see that script args and instructions are in displayed as arg - instruction - arg - instruction - etc. This makes sense in terms of usage in game code and it also makes it much easier to read them and to parse them:

Regarding udpating this schema, Godbert will still be able to display data so handmade updates will be possible, an update tool is WIP (https://github.com/xivdev/EXDTools/tree/updater) just like we used to have with SaintC schema so I could update most of it automatically. Worst case, just a PR to the new schema will make it work, you can also fork it, etc.

[slotthhy]

All correct. It might look similar or like the format has just been changed to YAML for the hell of it, but the overall ability to define structures rather than "columns" is significantly better than in the SC schema. No more parallel arrays because the schema can't support it, or ridiculous nested repeating groups when the actual data is just an array of structs.

As for updating, yes, it is manual still. The validation step that takes place upon creating a PR to the EXDSchema repository will verify that there aren't any glaring issues, and is required to pass. There is existing work on using column similarity to determine shifts, new columns, and removed columns, but applying this to EXDSchema and adjusting array sizes, structure sizes, and more is significantly more difficult than I anticipated. I've been asking around for assistance on that front since almost everyone who has used EXDSchema either via consuming it directly, using it in the Godbert build in this PR, or via Lumina's new sheet API has enjoyed it, and I don't want it to be a pain to update indefinitely.

In lieu of an updating algorithm, for now, I have no problem spending the short amount of time necessary on patches to get the definitions sorted out.

For a concrete example, take a look at SC SpecialShop vs EXDSchema SpecialShop

[slotthhy]

Hello! With Dawntrail, the SaintCoinach schema will be updated. I can take care of a lot of it.

EXDSchema will also be updated. The removal of the SC schema entirely will be delayed significantly while improving the functionality and code generation.

[ClayLivince]

So glad to hear that..... being desperate when updating Garlandtools after my SSD exploded and take my code and mysql db away.. Thank you!!!!