TeamSpen210
Can't parse TF2 static prop format
On the current dev version (hammeraddons commit cd80904 and srctools commit https://github.com/TeamSpen210/srctools/commit/885353c8c26ef6dd35a349f4a1d70fe9d9e6377d), I've been getting an error where a prop's flags are not being set correctly, and are causing the postcompiler to fail. Here is the error:
[I] propcombine.combine(): Propcombine sets present (13), combining...
[E] logger.except_handler(): Uncaught Exception:
ValueError: 294876348416 is not a valid StaticPropFlags
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "srctools\scripts\postcompiler.py", line 241, in <module>
File "srctools\scripts\postcompiler.py", line 162, in main
File "srctools\compiler\propcombine.py", line 898, in combine
File "srctools\bsp.py", line 644, in static_props
File "enum.py", line 360, in __call__
File "enum.py", line 685, in __new__
File "enum.py", line 668, in __new__
File "enum.py", line 813, in _missing_
File "enum.py", line 828, in _create_pseudo_member_
ValueError: 294876348416 is not a valid StaticPropFlags
For some reason, this error log that i grabbed did not list which prop, but previous ones did, and it was always the same prop (2fort_props/miningcrate002) and same flags.
Aside from hammeraddons, this should be all the potentially relevant info
- Game is TF2
- Hammer++ (should not affect bsp file at all)
- Stock compilers aside from vrad, which has the thread patch
- VMMC (vmm collapser), which does not cause the issue because my standalone vmf instances also have this issue (with different props, likely just due to which one is loaded first)
The collapsed VMF file is attached, it only uses default TF2 assets. koth_sawmill_omp.zip
Ah, I didn't implement support for TF2's special static prop variant - which has the additional section with lightmap scale values. It's a little tricky since it doesn't use a unique version number...
right, i'm not super knowledgeable with this stuff, but would a good starting place be checking for those flags and then assigning a version number based on that?
I messed around in the code for a bit, but couldn't get a working solution. I think a large part of the code will need to be rewritten because TF2's prop format moves flags to the bottom. I'll try and figure something out and submit a PR.
According to BSPSource there's actually several other minor game-specific alterations. I'll need to do what it does and check the total size of the data block to identify the correct one to use: https://github.com/ata4/bspsrc/blob/master/src/main/java/info/ata4/bsplib/BspFileReader.java#L314
that'd probably be a better idea than what i was trying, which was checking prop lump version and bsp version to see if it was tf2 (luckily its the only one that is v10 in bsp version 20). Should i continue experimenting or should i just leave it to you?
what a coincidence that this issue got mentioned recently! I recently started trying to figure this out again and managed to write an at least functional solution for the most part. I might end up making a PR in the future, but it will likely require a complete rewrite of the static prop lump parser to use prop lump version, prop lump size, and Steam appid.
Steam appid is needed because there are some games (like Black Mesa) which have different data but the same size and version (Black Mesa uses version 10 and is 72 bytes, however its based on CS:GO's version 10)
And since I can't figure out how to import appid, here's what I manged to do in case someone smarter knows how to implement it better.
Aside from adding the extra keyvalues, I calculated the size of the static props with this
distance_from-start = ((128 * len(model_dict)) + (visleaf_count * 2) + 12)
data_remaining = len(data) - distance_from_start
prop_static_size = data_remaining / prop_count
Essentially taking the size of the entire lump, and then subtracting it by the size of the model dictionary (each entry is 128 bytes), the size of the visleaf dictionary (each entry is 2 bytes) and 12 bytes (horribly hardcoded for now) from the count of models in the dict, visleafs in the dict, and props. From here, dividing the amount of data left in the lump by the amount of props gets us the size of each prop.
In the case of TF2, the size of each prop is 72 bytes, with a few interesting quirks. Despite being version 7 (at TF2's launch), TF2's static props do not have the per-instance color data found in other version 7 games like L4D. Alongside that, now the game calls it version 10, even though the format hasn't changed (meaning it doesn't have version 8-10 data) The most interesting quirk being that flags are now an integer and moved down to the bottom, HOWEVER, there is one VERY IMPORTANT THING. despite the flags being moved, the char where flags used to be is still there, leaving a null byte. This ends up making it pretty easy to implement, because the parser and exporter can simply run after the prop lump version 6 parser and exporter.
As a side note, for my implementation I added the size of each static prop to the StaticProp class so that it could be easily used for the if statements in exporting as well.
The actual format for TF2's static props ends up being something like this:
Vec Origin
Vec Rotation
UShort PropType
UShort FirstLeaf
UShort LeafCount
UChar Solidity
UChar Null (flags used to be here)
Int Skin
Float MinFadeDistance
Float MaxFadeDistance
Vec LightingOrigin
Float FadeScale
UShort MinDXLevel
UShort MaxDXLevel
UInt Flags
UShort LightmapResX
Ushort LightmapResY
Yep, that is the fix for the props lump itself, and what BSPSource does. I got sidetracked though trying to figure out the LZMA compression most of TF2's maps had as well.
Ah, fair enough, If I can figure out how to import the appid from game.py (I'm still learning python), I'll submit a PR for it. Although my current implementation works for TF2, its rather hack-y and would break games like Black Mesa and i'm not comfortable with creating a PR that knowingly breaks something
For the appid, you can't really, it'd be better if you add it as a parameter. The bsp module isn't specific to the postcompiler, it should work with any BSP you give it without knowing the game info. Perhaps we should add additional versions to the versions enum at the start, which you specify for games to override the behaviour in these cases?
Actually I had most of this implemented, static_prop_ver has functioning parsing of the props at least, haven't done writing yet.
It's definitely almost there! I've been doing a lot of testing during the past few days to figure out the cause of some of the issues that still exist and I think I've found them. These might not be TF2 specific, but I haven't tested on other games.
- The game crashes when opening the map. This appears to be an issue with how files are getting packed in, as it doesn't happen with --nopack (although does happen with --nopack and --propcombine). Interestingly, bspsrc still manages to be able to decompile this BSP, which then works in game when compiled! Its a very roundabout way of compiling a map, but it does work.
- comp_propcombine_set entities are still in the BSP after the postcompiler runs. I'm honestly not super sure why this happens, since the postcompiler entities do get removed in other games, to my knowledge. Is it possible that this lump is different too, and that we're deleting too much/little?
I also have been having some issues with props not getting added to a combined prop, but i'm willing to write that off for now.
Well propcombine packs the combined models always, overriding the --nopack option (so the pakfile will still be modified). The pakfile is a unique lump with some special rules - it never gets compressed unlike all the others which is a bit tricky. Are the propcombine entities still in the BSP with --propcombine passed, or without? The removal code is different for each, so I might have missed one.
No matter what, the entities are still in the bsp. Just tried it with only --propcombine and they're still there.
Honestly, after doing more research, I'll write off the crashing issues as being due to the LZMA compression, as the only differences I can find within the pakfile lump is:
- version needed to extract (postcompiler returns 10, vbsp returns 20)
- date and time modified (postcompiler returns actual data, vbsp returns null)
- some files are in a different order
Despite these differences, TF2 seems to handle it just fine?
The entities not being removed is likely also due to that.
Guess this goes to show that I really should do a bit more research before posting anything, sorry!
I should've closed this a long time ago since this has been fixed for probably over a year