File Duplicates from updates only load oldest version available

[Lonemaster]

Game

Borderlands 3

Error

[No Error log]

Reproduction steps

I am unfamiliar with other Unreal Engine games, so I will briefly explain the way that BL3 does its updates, in case that it is unique in that way it handles file updates:

Whenever a file in the game (ex.: A DataTable to read out health values for a group of enemies) is updated, instead of forcing the player to re-download the entire .pak, a duplicate is created which contains the updated file, with the same file path, where the game loads the most recent one available.

The issue is that when it comes to these files, FModel 4.0.1.1 and 4.0.2 always load the oldest available of these files, regardless of which of the duplicates is selected. The earlier versions of FModel 4 acted the other way around, only loading the newest of the duplicate files.

The last version of FModel 3 (3.1.1.1) that was able to load in files from Borderlands 3 did not have this issue. For example, the file OakGame/Content/Enemies/EdenBoss/_Shared/_Design/Balance/Table_Balance_EdenBoss_PT1 has one update-duplicate, which, in FModel 3.1.1.1 displays Edenboss to have a HealthMultiplier of 200 in the pakchunk4-WindowsNoEditor.pak file, and a HealthMultiplier of 160 in pakchunk4-WindowsNoEditor_16_P.pak, whereas the latest FModel 4 version shows 200 in both files. This issue is not specific to this file, or the DataTable file type, as it also extends to OakGame/Content/Gear/Artifacts/_Design/PartSets/Abilities/_Legendary/Misc/VictoryRush/Ability_Artifact_VictoryRush and others.

[Lonemaster]

Apologies if this is formatted incorrectly, I am not familiar with how things are done on github. If there are questions, I will answer them to the best of my abilities.

[Amrsatrio]

I understand what you mean. FModel's asset reader was not made for this. A fix will come later.

[4sval]

I didn't understand everything and I can't decide whether it's a smart or dumb way to update / hotfix a game. How does the game know what's the most recent archive?

@Amrsatrio I think we should start loading packages from the ui by their id

[Lonemaster]

As far as I can tell, the only way to discern which version is newer without knowing the value it was previously should be to look at the _pak file that it came inside of. The paks are ordered in versions, which are stated directly in their names; pakchunk0-WindowsNoEditor is followed by pakchunk0-WindowsNoEditor_0_P , which is followed by pakchunk0-WindowsNoEditor_1_P, and so on, all the way to _17_P

As for whether it's smart or dumb, I can't tell. The most likely explanation I have been able to come up with is that it is easier to let the players download new paks with the relevant files, rather than having them redownload the entire thing. The upside is that update download sizes stay low, but the downside is that since old versions do not get deleted from the pak files, the game's size slowly bloats up more than necessary. A fair amount of Borderlands 3's files exist over ten times at once because of this.

[ryantheleach]

+1 Sea of Thieves also updates in this manner. Previously every update would be 50gb, then they swapped to this method when they discovered that having multiple hotfixes a month, at 50gb each was putting a strain on both their bandwidth, and users patience.

I assume they periodically condense them down when it gets out of control.

[nickgeiserr]

why would a game fully redownload a game? just download the parts that need updated...isn't that what multiple pak files is for

[ryantheleach]

But usually you'd expect the pak files to replace the existing ones, not complement them with duplicates inside.

However games still do it.

[Kein]

But usually you'd expect the pak files to replace the existing ones

Only for the projects that were made to with granular pakchunks from the start. You have to understand, trying to sort out your assets to each chunk AFTER you already developed project for many years as one big package is close to impossible, it is a herculean task with thousands of dollarinos needs to be spent on testing and catch gazillions of new errors and issues. It simply not worth it. And nobody really capitalizes on chunk separation properly so there you have it. Great features left unused.

[PaperStrike]

The code responsible for this actually lies in the unreal engine. It's not so game-dependent.

1. FPakPlatformFile::MountAllPakFiles (permission required) is where the engine finds and mounts the pak files.
2. FPakPlatformFile::Mount (permission required) is where the engine decides the priority of these files, where line 7239-7265 specialized for _P chunks.

There are much more games I suppose, especially new games, use pak files like this. Engines before 4.17 don't have the _P part so detailed and thus buggy when there're more than one patch chunks - games released before 4.17 should have developed their own ways to patch - likely to be the reason why games like Borderlands 3 patch differently with Fortnite.