Source-1-Games
Source-1-Games copied to clipboard
ValveSoftware
Can Source games not share HL2 VPKs?
At the moment, Half-Life 2, Portal, HL2DM, Day of Defeat and Counter-Strike all have a directory named hl2 containing identical VPK files for a total of just under 3GB. These VPKs in each game's hl2 directory has the same checksum and thus content.
This is a little inefficient, isn't it? I understand that when we used the GCF format, these games did share resources but SteamPipe's greater streamlining of game packaging and distribution has given way to an additional 3GB on the download of HL2 engine games.
In my experience, deleting the VPKs from Portal and symlinking them from Half-Life 2's own installation folder works a treat. Steam doesn't overwrite them on cache verification and the game works perfectly fine. The same is observed in Day of Defeat, Counter-Strike and HL2DM with a space save totalling just under 9GB.
I suppose this gives rise to a few questions. First of all, does Portal need those VPKs in the first place? I mean, they contain audio files of Eli talking about Dr Breen. Eli and Dr Breen don't appear in Portal. The same goes for the other games. Secondly, could SteamPipe/Source allow for shared resources like this? Could we actually install Portal and others as HL2 mods like Episode One and Two are (SteamApps/common/Half-Life 2/portal)?
In many environments an extra 10GB of disk usage isn't really a problem, but it seems odd that the otherwise streamlined SteamPipe migration has given way to this little inefficiency. I suppose I could be nitpicking but I am genuinely curious about why we've ended up with 5 instances of Half-Life 2 resources.
Exactly my problem too. The GCFs could be shared, but now each game has its own HL2 VPKs.
If the games can't share the VPKs, at least have them store what they just need. Portal needs the HL2 sounds for the ambient soundscapes, but it doesn't need the Barney dialogue and the Alyx face textures. This inefficiency must be solved, because SteamPipe was being advertised that it would "lower each game's space on the disk", and instead users need to have more disk space because of a forced update.
I don't know about "must" be fixed, there are bigger fish to fry (like all the Orange Box bugs! Come on, make Breen stand up and fix the lip syncing, dammit. It's been like 3 years!). But yeah, it could potentially leave a few people a bit disillusioned about SteamPipe.
Yes, I think that the botched HL2 engine upgrade is more immediate. All those little bugs (including Breen not standing up, HL2_song27_trainstation2 not playing) and bigger bugs like the stretched skyboxes completely ruin the atmosphere. It's unfortunate for both old fans who can't relive the awesome experience, and new players who will never get to play that brilliant 2004 game.
But the HL2 VPKs have to be fixed at some point too. SteamPipe is supposed to lower the disk space of games, and it actually increases it.
I don't remember SteamPipe ever being said to "lower disk usage". All I ever really heard about it was that it was a "more efficent format". The only games I've seen to share VPKs are HL2 and its expansions. That includes custom files.
The SteamPipe help page used to say that TF2 will use about 1GB less after the conversion. That was removed though.
But still though, because this is a forced update, this means that anyone who has lots of Source games installed is forced to have their games taking even more space. I can't see why the VPKs can't be shared.
It's a strong matter for me, too. I have a HDD that isn't much large, unfortunately, and it is equally divided by Linux and Windows partitions. On Linux partition, I have installed many Valve games such as Portal, TF2, HL2 (only main, not the episodes) and L4D2 Beta, and some indie games. All those games plus other software installed leaves me really few free GB of space in the partition, that means if I want to play the HL2's episodes or any other big games, I'm forced to uninstall one of four Valve games, because they aren't sharing the same engine's files, resulting in a lot of wasted GB that could be used for other games. I know, I could uninstall Portal and HL2, but this doesn't resolve the issue anyway. I hope it's getting fixed soon :)
Yes it just occurred to me that TF2 also contains these files. So we're now over 10GB of (potentially) unnecessary space usage.
ALL Source games have those files. So that means 2.6GB more for each game.
What's especially hilarious is that 90% of the content of these originally-shared directories isn't actually used in the games; the files are solely there to waste space and (in the case of TF2) trip up newbie custom map makers who don't realize they're from the wrong game. And trip up Garry's Mod with the occasional identically-named files getting substituted for each other. If stripping them out would solve all three of the above problems, I say they should go for it.
Personally, I just used a program in windows that creates junctions (similar to symlinks in linux), so I just have 1 hl2 folder at Half-Life 2/hl2 and all the other ones are junctions. This creates a problem though: sometimes steam decides that it should re-download the vpks for one of the games and this can break the others. For example sometimes it downloads the files for Portal, which don't have some loading material that is needed to start CSS, so it crashes.
I think it should be possible to just have one set of hl2 vpks and have the other source games mount it. Perhaps these could be stored inside a 'Source Engine' folder with all the other games as 'mods' so as to not have games like Portal be in the Half-Life 2 folder. Or other games could just mount the vpks straight out of that folder without needing to move anything.
@niobium93 my comment here https://github.com/ValveSoftware/Source-1-Games/issues/213#issuecomment-19131552 says why they choose not to share the files, and your first paragraph highlights why they don’t, #213 got closed because its unlikely to change in the near future.
This is no excuse for not having shared VPKs. Tea23 managed to get them shared across the games.
Besides, if the games are totally independent, then what's the need for Barney dialogue sounds and Alyx face textures in the Portal files when they don't get used at all?
Yes, if sharing them could cause problems, then Valve should do what DigitalStuffMan suggested; clean the files to remove unnecessary content.
IF they decide to fix it. In #213 they didn't seem that determined to find a solution.
http://store.steampowered.com/news/9951/ included "support for better content sharing between games", so it is possible. Presumably Valve's strategy is instead to clean out all the unused files at some point.
Come to think of it, Left 4 Dead 2 should share the voice clips and music from Left 4 Dead for its reworked campaigns. They take up a lot of space. Especially given that they're mostly uncompressed WAVs.
Half-Life 2 Singleplayer does this by placing the games inside Half-Life 2, but why not a Source Multiplayer directory for hl1mp, hl2mp, cstrike, dod, tf?
So, whatdy'all say of my idea I brought up? A Source Multiplayer directory in the common folder (like with Half-Life 2) so all the Source 1 Engine Multiplayer games go in that directory? Saves even my ass from having duplicate HL2 content in the games instead of almost everything packed together until needed.
Those Synergy folks sure knew what they were doing when they converted to SteamPipe, their Gameinfo.txt points towards the Half-Life 2 directory and its resources! Sad part is, HL2 and its episodes need to be in the same area Synergy is installed in... SearchPaths { // Synergy paths game+mod+game_write+mod_write+default_write_path |gameinfo_path|. gamebin |gameinfo_path|bin game |gameinfo_path|synergy_pak.vpk
// Required Half-Life 2 content (assumes default install locaton, this section needs to be edited otherwise)
// These are relative paths from this file (gameinfo.txt), to the required files.
// If you edit this for a non-default install location, please keep a backup of your changes for this file may update again.
platform "|gameinfo_path|../../Half-Life 2/platform/platform_misc.vpk"
game "|gameinfo_path|../../Half-Life 2/hl2/hl2_misc.vpk"
game "|gameinfo_path|../../Half-Life 2/hl2/hl2_pak.vpk"
game "|gameinfo_path|../../Half-Life 2/hl2/hl2_sound_misc.vpk"
game "|gameinfo_path|../../Half-Life 2/hl2/hl2_sound_vo_english.vpk"
game "|gameinfo_path|../../Half-Life 2/hl2/hl2_textures.vpk"
platform "|gameinfo_path|../../Half-Life 2/platform"
game "|gameinfo_path|../../Half-Life 2/hl2"
// Extra content for Hammer
game "|gameinfo_path|../../Half-Life 2/ep2/ep2_pak.vpk"
game "|gameinfo_path|../../Half-Life 2/episodic/ep1_pak.vpk"
//game "|gameinfo_path|../../Half-Life 2/lostcoast/lostcoast_pak.vpk"
//game "|gameinfo_path|../../Half-Life 2/hl1/hl1_pak.vpk"
//game "|gameinfo_path|../../Half-Life 2/hl1_hd/hl1_hd_pak.vpk"
// [Add any additional content for Hammer to mount here]
// downloaded content here
game+download |gameinfo_path|download
}
}
} [/CODE]
Xyzzy_Swe happens to have a guide for this too!
i find it that it looks kinda odd, but i get why it is there, for example even a single instance of hl2 texture/sound being present in those games needs complete testing and thus it is easier to just buckle it up with all of hl2 content, although hl2 voice overs in those games do question why it wont be simply a hl2 bin vpk with hl2 engine content stripped to bare minimum to run those games. in my case i just really don't like that user made content that clearly is badly designed, using hl2 content in games like tf2 and stuff i find it frustrating to walkthrough this approach were a game is basically allowed to be as broken and as uncohesive as possible where you play on a counter-strike server that looks like a cartoon show, same with this treatment of content as if we all need hl2 content in every game just in case 2 broken maps miss hl2 content in them. i think all source engine games need a serious uplift were it works through workshop and if there is a case of hl2 content missing, those maps should be patched instead of carrying the weight of entire game just so that we can retain compatibility with those few maps that do rely on it.
I looked through the folder structure of all the first-party titles, and I think I came up with a comprehensive list of which games have duplicates of other games' VPKs.
Uses Half-Life 2's hl2 folder, because it installs in same folder
-
Half-Life: Source (
hl1) -
Half-Life 2 (
hl2) [naturally] -
Half-Life 2: Lost Coast (
lostcoast) -
Half-Life 2: Episode 1 (
episodic) -
Half-Life 2: Episode 2 (
ep2)
Has duplicate hl2 folder, because it installs in a different directory from Half-Life 2
-
Counter-Strike: Source (
cstrike) -
Day of Defeat: Source (
dod) -
Half-Life 2: Deathmatch (
hl2mp) -
Portal 1 (
portal) -
Team Fortress 2 (
tf2)
Has duplicate hl1 and hl2 folders, because it installs in a different directory from both Half-Life 2 and Half-Life: Source
-
Half-Life Deathmatch: Source (
hl1mp)
Additionally, most mods for Source Engine and GoldSrc games seem to have their own duplicates of any game dependencies, which makes sense for Half-Life 2: Update, but is pretty wasteful for most other instances. Synergy seems to be the exception to the rule, as @BreakinBenny mentioned… seven years ago.
While I understand why Portal 1 might need to have separate binaries, I fail to comprehend a potential reason why the multiplayer titles couldn't be merged into one directory with one executable (or multiple renamed ones), like Half-Life 2's singleplayer titles do. The GoldSrc titles do the same thing; Ricochet and Counter-Strike: Condition Zero use the same binary and steamapps\common directory as Half-Life 1 and Team Fortress Classic.
Please at least strip the individual games' duplicates of the VPKs down to only the necessary files; the wasted space discourages users such as me with limited drive space from playing these older multiplayer titles.
@SirYodaBelt I know for a fact that by making HL2DM's Gameinfo.txt point towards base HL2 VPKs, the menu and in-game look messed up because they're trying to find some graphics that are missing in the "new" ones... and vice versa.
Perhaps if the games altogether were located in one folder, call it "steamapps/common/Source 1" for instance, then we can have TF2 freely utilize the base Source content and textures as it sees fit, as can HL2DM. (I'd hope that this can also be made possible even if someone has one Steam library on each hard drive, like my C: and D: drives.)
Heck, old games like Return to Castle Wolfenstein on id Tech 3 use "mp_*" when to fetch something for multiplayer, "sp_pak#" for singleplayer (like those textures for Nazi imagery if you weren't playing the censored German version), and "pak#" for "at all times".
I know for a fact that by making HL2DM's Gameinfo.txt towards base HL2 VPKs, the menu and in-game look messed up because they're trying to find some graphics that are missing in the "new" ones... and vice versa.
This difference in VPK contents probably has to do with the fact that singleplayer and multiplayer Source 2013 games are technically on different engine branches.
I think the optimal solution here would be to install the common/Source SDK Base 2013 Multiplayer/hl2/ folder on each drive that has a Source 2013 MP game installed, then edit the gameinfo.txt for all those games (CS:S, DoD:S, HL2:DM, HLDM:S, and TF2) to point to that shared location and remove their local hl2 folder. Same with Portal 1, except that should point to Half-Life 2's hl2 folder.
I believe this is the optimal solution, because moving all the games into one location would likely break a lot of peoples' user configs and mods, similarly to the SteamPipe update; this solution is more error-proof. Additionally, having separate common/<exedir>/bin/ folders for the different games is a bit more convenient for modding, as you don't need to select a game configuration every time you launch Hammer, and you can have stuff like the build of Hammer++ that is specifically designed for TF2.