Feature: support for behavior mods

[rfortier]

Behavior mods such as those generated by Nemesis or Pandora.

Important note: this PR supports conditional compilation, which is currently set to off in the last commit. This was done to enable the code to merge without risking the imminent release. If the conditional compile is switched on, the support will be enabled in the clients and those clients should be able to use the same servers as clients without the support.

Goals

• Not risk the release.
• Enable compilation of a "pre-release" or test version that could be compatible with the public game servers.

Prerequisites

• Previously, PR #658 was a prerequisite. Cosi asked me to merge that branch, so this is no longer true.

Testing

Four combinations were tested to ensure the PR is safe:

1. Compiled off, no mods: animations sync as expected.
2. Compiled off, with behavior mods: behaviors not synced, characters skating, as expected.
3. Compiled on, no mods: animations sync as expected.
4. Compiled on, with behavior mods: behaviors sync for the mods supported in the PR. Support for additional simple mods that just need an added variable or two synced can be added by dropping the definitions into the SkyrimTogetherRebornBehaviors directory.

Theory of Operation (how it works)

It's a big PR in line count, but much of that is machine-translated code. The major topics to cover are:

1. Directory layout, source directories and installation directories.
2. How the mod accomplishes its goals of enabling simple animations (no complex event management or scripting) generated by Nemesis or Pandora to sync.
3. Approaches to enable STR devs and modders, even end users to make behavior changes without stepping on each other.
4. Support for syncing additional behavior variables (in particular, more than 64 boolean variables, of which the STR code is already using 62). Note this requires a protocol change between the server and clients, so this code always compiles, while the remainder is conditionally compiled. This approach simultaneously supports two goals: the STR team can release without the mod support for lowest risk, but a version not supported by the STR team can also be compiled, and use the same servers.

Directory Layout / Changes

Source Directories

• One source directory is added to the source tree at client\ModCompat. The changes support both Skyrim and Fallout4.
• One file, an encumbrance, is added at \Libraries\magic_enum.hpp. This enables translating enum names to their string form, despite C++ not supporting reflection (yet). It is governed by the MIT license.
• A directory tree is added to \Gamefiles\Skyrim\SkyrimTogetherRebornBehaviors. The files in this directory enable the STR mod to sync additional behavior/animation variables if a mod requires it. Any variables named in the tree are merged with the base variables the STR dev team defines. Additional files define the original STR hash of behavior variables, and a signature, a unique list of behavior variables that must exist (or must not exist) to uniquely identify an Actor.

Installation Directories

• The source directory \Gamefiles\Skyrim\SkyrimTogetherRebornBehaviors must be installed alongside the SkyrimTogetherReborn directory by any FOMOD or 7z installer. It is deliberately outside the SkyrimTogetherReborn directory so 3rd party tinkering is outside of the game directory.

Theory of Operations

In a sentence, the mod works by detecting AnimationGraphDescriptor hashes that don't exist in base STR, seeing if the Actor's animation variables match a signature in the SkyrimTogetherRebornBehaviors directory, and if so, merge the STR variables with the ...Behaviors directory variables (if any, it might just be the hash was changed), and generates a new AnimationGraphDescriptor with the new hash.

The interception requires a few lines of code in References.cpp to intercept where animation variables are prepared for serialization and deserialization. They call out to BehaviorVar.cpp:BehaviorVar::Patch() when the AnimationGraphDescriptor lookup-by-hash fails.

::Patch can fail and return NULL again. This is normal; it happens when STR doesn't need to sync behaviors, syncing position and velocity is enough, so there is no AnimationGraphDescriptor at all for that creature. Wisps are an example.

If a hash lookup fails, it is added to a failed list and ignored for 10m. This because searching for a signature is semi-expensive, but more because it generates a ton of logs that aren't useful which is the real cost. 10m is long enough for most encounters to end, but short enough to try again if dynamic behavior modification is ever properly supported.

Intricacies of Merging Behavior Var Lists

Nemesis doesn't just add more behaviors to the end of the list; it rearranges the list. That's unfortunate, and it is why humans skate if just Nemesis is installed without adding any more variables to be synced; The rearranged list has a different hash.

To accomplish a merge, these operations occur:

1. We must be able to translate the base STR AnimationGraphDescriptors back to their string form. This is done at authoring time by running a script that translates the AnimationGraphDescriptor* files to a form that magic_enum can use to generate the string form of each enum, for everything in the Skyrim and Fallout4 directories. Currently, the output files are checked into Git. It would be better to get the xmake dependencies right, but I'll need some advice for that. And it can wait until after release.
2. At game startup time, the SkyrimTogetherRebornBehaviors directory is scanned. Each subdir is expected to have a *hash.txt file with the original STR base game hash, and a *sig.txt file that contains a list of behavior variables that must and must not exist to uniquely identify the Actor. Additional *bool.txt, *int.txt, and *float.txt files name any additional variables that must be synced. Each mod author can have their own files with a unique prefix, they are all merged and stored in a list of BehaviorVar::Replacer objects.
3. At run time, the modded behavior is matched to a Replacer via the unique signature.
4. The Replacer identifies the matching STR base game AnimationGraphDescriptor, which is translated back to strings which are stored in a ::Set for each of bool, int and floats.
5. Added variables (if any) found in the BehaviorVar::Replacer built from the SkyrimTogetherRebornBehaviors tree are merged (uniquely, because of the way ::Sets work).
6. The sets are translated back to an enum form to construct a new AnimationGraphDescriptor.
7. That descriptor is added to the game data structures in AnimationGraphDescriptorManager::Register

From that point forward, the new hash will be successfully found by the STR base-game logic.

There are a few details that added some complexity. In particular, there are misspellings of variable names even in vanilla Skyrim, usually incorrect case like speed when the correct variable is Speed; that happens with a particular Winterhold guard, for example. Having no way to know if Skyrim ever uses any variables only distinguished by capitalization, the code tries a few common variants if a lookup fails and generally will correct the problem.

Decoupling STR dev team and Modders

All this logic to match and merge is to ensure the STR dev team and modders can make changes independently without interfering with each other.

Support for Syncing Additional Behavior Variables

All of the above support will ensure animation sync if Nemesis/Pandora are used, and if no new variables need to be synced will even be compatible with the same servers used by clients without this animation support.

But to also support mods that do need to sync additional variables, such as the popular True Directional Movement, this mod needs support for syncing more variables. In particular more Boolean variables, STR base uses 62 out of a limit of 64. The limit is baked into protocol between the clients and server, so lifting the limit creates a compatibility problem.

This PR includes a merge of branch feat-unlimited-behavior-vars, which always compiles once this PR is merged to ensure versions compiled with behavior mod support will still be able to use the public servers of the same version.

Rather than simply turning the boolean long long in the code into an array, this code changes the definition AnimationVariables::Booleans from a uint64_t to a TiltedPhoques::Vector< bool >. This compiles into an array of bits with the current tech stack (this isn't guaranteed by the C++ standard, but is the norm).

This design choice was made because:

• It actually takes 1 byte less on the wire than the than the approach of just adding another uint64_t in the "large" case of sending humanoids.
• It takes less space on the wire than the former protocol for most creatures that don't sync very many variables.
• Consistent code, everything is a Vector now.

[rfortier]

Some more info for people interested in testing this, but not wanting to bother with compilation.

Please report issues here on github, not to the STR dev team. Or find Ujave on the STR Discord.

I have compiled up two versions of this mod and released it on my github; a version with the compilation flag turned on, and a version with it compiled off for testing that it is safe. There's a FOMOD installer provided so anyone can try them. You can find that here.

There's not a lot of documentation yet about how to support additional mods, but @MostExcellent or I can help.

[rfortier]

As requested, I merged the content of PR #658 into this PR.

Support for behavior mods is compiled off by default. The former content of PR #658 always compiles.

I'll close PR #658 now.

[RobbeBryssinck]

What's missing from this PR is a general overview of the PR's contents and how you've solved the problem at hand. Currently, I'd have to read the code to do that, which is a bit much given the size of the PR.

[rfortier]

Rebased to /master for release.

[rfortier]

Updated branch to the latest /dev.

It currently compiles with MODDED_BEHAVIOR_COMPATIBILITY undefined. So only Unlimited Vars is compiled, making a standard build server-compatible with a modded animation build.

Revert the last commit to build an animation build.

[rfortier]

I'll run through your review comments. Tried to follow the style guide, but at least missed capitalized funcs/methods. Time to figure out the clang formatter, I guess.

Looks like a VS reinstall lost my tabs->4spaces setting. Is that the source of the generic "spacing" comment? And will clang formatter fix it? Otherwise I can do it manually.

Looks like I missed converting some contributed code to TiltedPhoques::. But one is deliberate; if I remember right TiltedPhoques::Map and ::Set are hashmaps, and I needed a sorted map / set; is there one?

Found TiltedPhoques::SortedMap, but it doesn't look like there is a sorted set.

Performance isn't super-critical for these, as they are only used once per creature type per launch (as in, once for all humanoids, once for all dragons...). So even if TP::SortedMap is slower, it won't hurt anything.