iesdp
iesdp copied to clipboard
Gibberlings3
Machine readable format
I'm thinking that it could be beneficial to have IESDP data in machine readable format. Currently there are multiple tools all parsing and reading IE files on their own: DLTCEP, NearInfinity, iesh, probably others. Inevitably, differencies and inconsistencies arise. But if file formats were described in a structured way, we could have a single, truly definitive source of information, and pull updates from it (semi)automatically. (Which is my ulterior motive for BGforge MLS.)
As an example, take a look at sfall docs: functions listed in a yaml file, with some python scripting it converted to markdown, and markdown is published with jekyll, resulting in a nice site.
If you look at opcode description, it's already basically yaml, with html additions. Could probably converted into true yaml semi-automatically.
To be clear, this is not about IESDP looks, just internal data representation: binary file formats, script actions/triggers, effects.
Tagging @Argent77, @4Luke4, @AvengerTeamBG, @FredrikLindgren, @ALIENQuake.
Something like this might have been useful 15 years ago, but now that almost everything is already written, I don't see anyone gaining anything by rewriting all the parsers, writing generators and rewriting the way the data is stored.
On the IESDP side of things the last part may still happen though, since it could then convey the data better (eg. #6 #19 #26).
almost everything is already written
MLS isn't, hence this issue. Obviously, I can't speak for devs of other tools, but MLS would benefit.
@lynxlynxlynx let me put it like this: if a such pull request was coming your way, would you be opposed to it?
Depends on what everything you'd stick in it. Eg. if you want to make the file formats machine readable, that's great and wouldn't need to affect deployment, since jekyll has the needed support: https://jekyllrb.com/docs/datafiles/
Do you want to define a format for, say, opcodes? Or I should do that myself?
I don't know what you mean, since the opcodes are already easily machine readable. Everything is parametrized and the description is plain html most of the time (a few liquid tags here and there).
Well, html with liquid parts is not exactly machine readable. Still being parametrized is exactly why it's easier to start with them.
To make an example, maybe something like this
_data/opcodes.yml
- n: 0
name: AC vs. Damage Type Modifier
type: stat
param1: AC Modifier
param2: Type
bg1: 1
bg2: 1
bgee: 1
iwd1: 1
iwd2: 0
pst: 1
doc: |
Applies the modifier value specified by the 'AC Modifier' field to the category specified
by the 'Type' field.
Known values for 'Type' are:
- 0 All
- 1 Crushing
- 2 Missile
- 4 Piercing
- 8 Slashing
- 16 Base AC setting (sets the targets AC to the value specified by the 'AC Modifier' field.
If the targets AC is already 'AC Modifier' or below, this effect will do nothing).
Each modifier type to AC from this opcode is capped to the range [-20, 20]. Each AC type total is capped to the range [-32768,32767].
notes:
- |
IWD1 and PST use a slightly different version. The "Base AC" sets to **field - 1** instead.
IWD2 uses different parameters altogether.
The notes are sometimes interspersed in the description, not always at the end. And there can be several of different severities, so the proposed format is not good enough. I also see no way to avoid needing to clean up the descriptions on the user side of things, eg. to get rid of broken links.
And I'm definitely opposed to cramming them all into one file. That's a simple step users can do if they need it.
The "Known values" can also be one of yaml key with values, right?
Right, but I'm not sure if will be possible to parametrize them and keep current html layout unchanged. And I don't know how I'd use this data yet. If anyone needs it, they are welcome to chime in.
The notes are sometimes interspersed in the description, not always at the end. And there can be several of different severities, so the proposed format is not good enough. I also see no way to avoid needing to clean up the descriptions on the user side of things, eg. to get rid of broken links.
Severity is easy to deal with, adding a separate stanza (warning, important) should be enough.
Being interspersed is harder. Not sure if there's a way to parametrize this properly. I'll search around, if not, I guess keeping the doc monolithical will have to do.
Could you clarify the bit about broken links? Which ones do you mean, why are they broken?
And I'm definitely opposed to cramming them all into one file. That's a simple step users can do if they need it.
Jekyll can read data from dirs, something like _data/opcodes/0.yml. Though it does start to sound like simply moving/renaming opcodes dir, but I'm trying to work out something that'll work for actions and stuff with minimal changes later.
Links: opcodes link to each other via anchors, so you'd end up with broken links unless you replicated the naming and therefore layout (all in one file).
Actions and triggers won't be much better; except for less data, the same problems apply.
Depending on how this works out, links might be kept and lead back to IESDP:
Maybe it's better to start with something else than opcodes, indeed. Here, I took a shot at actions. Sample data is stored in the data files. I wanted to avoid duplicating info (action numbers in both filenames and files themselves, etc), but liquid is not flexible enough, so the overall system is almost the same as opcodes, just doesn't require setting game to 0 to filter it out (can be just skipped). You can launch it locally, check out BG1 and IWD2 action pages.
Looks good. From what I can tell, action descriptions only use colors (can be replaced) and links (markdownify takes care of that?) on top of what's shown above.
Loaded BG2 actions. There's minor difference in styling, mostly because markdownify wraps everything into paragraphs. I made adjustments, but didn't go out of my way to make it exactly the same before getting feedback. (One thing I did fix intentionally is ugly codeblocks)
If all's good, next step would be to load actions from other files, run comm/md5 to find and delete the identical ones, then some diffs to find and combine those that only differ in wording, then move all the rest as variants.
Sounds good, but let's resolve #85 first, so it's clearer how some of the more interesting actions turn up (since you skipped a few).
Most for #85, but also 349 that doesn't have any colouring. Do something like find | wc -l in the dir and you'll see it doesn't match the max action id + 1.
These were missed due to a mistake in the script, now included. Count won't match anyway, since there are gaps (NID*). 349 is a special one, it has some formatting messed up, but I think it can be corrected latter, after adding variants, along with other manual cleanups here and there. The table in 349 is just styled as code. I think it's not worth to add a separate kludge just for it, considering that it looks fine, but if you think it's important to have the same background, I guess it can be done. Pushed updated data.
Well, they are counted as one "action" as far as Jekyll is concerned. Otherwise, they'd produce a full list of "not working" actions, so I thought to keep them combined. If everything's looks so far, please let me know, I'll proceed with variants.
It doesn't matter for the output, but for people like you that want it as data, it makes no sense to jumble them together.
I'm not sure what use they are, only completeness. Certainly no point in adding them to completion, everything not working will be skipped. So do you want to separate them?
Nothing comes to mind, except that let's do plain bg2 first and iron any problems out before continuing. Also, another PR is open that touches the ee action list and it would just cause conflicts if it wasn't merged first.
You mean merge upstream? If just BG2 has manual updates applied and merged, that'll make it a little harder to search for differences later. Also, there won't be links to variants, since that data doesn't exist yet.
One more thing I'd like to point out, currently action aliases are added to the same file, how's that?
Ok, then wait a bit, since @4Luke4 is almost done.
Aliases I don't like that way, since I think eg. the RES variants are present in some games without the default version, so it would complicate the layout logic to iterate properly. Just keep it KISS and create a separate file.
I would like to avoid duplicating descriptions in data and displaying duplicates too, but not sure how to do that while allowing for variants. I'll think about that meanwhile.
Edit: ah, current version doesn't have variant links too, so my second point about merging is moot.
Ideally the descriptions wouldn't be duplicates anyway, since they should explain the parameters used.