FAFReplay Format v2

[thygrrr]

Currently, gameinfos and replay infos are a bit of a quagmire.

We need to track:

• binary-patch hex + tag + repo url
• featured mods name + hex + tag + repo url
• sim side mods name + hex + tag + repo url
• map name + hex + tag + repo url

The repo url in every case needs to be verified: if replay pulls data from unverified locations, warn user and give a choice to abort.

We want to track:

• players
• factions
• player metadata
• duration
• outcome

Will be updated with a v2 format draft (suggestions welcome).

[Crotalus]

Does this mean that all sim-side mods uploaded in modvault need its own git-repository? Or is all those mods in the same repository and we somehow extract the correct commit for each mod?

[Eximius]

Version means the shasum hex and/or tag, repo url tuple.

I reluctantly agree that we need to track Engine version and main mod, sim mods and map versions in our custom FAF replay header. We should also track duration and if the replay is a complete version or not.

Related, we should agree on a convention for injecting mod versions into mod_info.lua or a custom 'mod_version.lua' so that mods have introspection from inside the game. Same for maps, main mod.

All other metadata should be contained within the normal Forged Alliance replay metadata, which is created by the game itself.

This way the FAF replay metadata would provide only the things we necessarily need for easier engine/mod/map versioning and the FAF patcher can just read only this section to know what to fetch, checkout. If we need the player metadata, mod decriptions, other, we should just parse the lua value contained within the FA replay header.

The FAF replay header will not be exactly streamable due to having to know the durationg and completeness of the replay. It will require a seek to the front after the replay is finished to commit duration and completeness values.

Technicalities:

FAF replay header has to have a "FAF_REPLAY_v2" tag or some other. Has to provide easy offsets to the FA header and the FA beat data. After the FAF replay header, FA replay header should follow unchanged. After this header, the beat information should follow zlib'ed to save space.

We should stream FAF replays to the server instead of simply FA replays. This is entirely possible as the server will figure out replay completeness as well as duration.

[faf5678]

the encoding shall be specified in the faf header. which would be 'base64' for the current format and 'raw' after you believe me that base64 is fail. also the compression is not compatible with gzip,compress,zip,bzip2,younameit. i can only reproduce with the python lib. a standard compress format would be better.

[Eximius]

It is standard. It is zlib'ed. The only difference is that it has a integer (4byte) at the start describing the size of the decompressed data before zlib compressed blob.

[faf5678]

ok you just need to prepend a gzip header to the zlib stream if you want to uncompress it without python:

tail -n +2 R.fafreplay | base64 -d | cut -b 5- | (printf "\x1f\x8b\x08\x00\x00\x0\x00\x00"; cat) | gzip -dc

[Sheeo]

Okay so, I think we should go with this in the replay header, as a general version descriptor for launching the game. Fields marked with ? are optional. Repo is short for reponame, and consists of a github-style username/reponame string, i.e.: "FAForever/fa".

binary-patch: {url?, ref, hash?} featured mods: [{url?, repo, ref, hash?}] sim mods: [{url?, repo, ref, hash?}] map: {url?, Friendly name, ref, hash?}

Replays/version descriptors missing any hash should be considered unstable, as they are "moving targets". The algorithm for updating the game would be, for each type of repository:

1. See if there's a url given. Validate that it's within our "trusted" set of URLs, if not: mark as untrusted.
2. See if there's a hash given, if so, fetch+check this out directly (Perhaps validate with given ref, but not needed)
3. See if theres' a ref given, if so, fetch+check this out and mark version as unstable.

This approach allows flexibility in that we allow external urls, and we allow to run a replay with different code than it was originally generated with. This can be quite useful for testing.

I'll code this functionality today or tomorrow, and we can do code review/further discussion :)

[thygrrr]

Suggestion: We'll allow any URL but the lobby client has a set of trusted remote roots for mods, maps and games. This should be user-editable eventually (lobby settings dialog).

[Sheeo]

Yup.

[Wesmania]

Is this still being worked on?