CUE4Parse

CUE4Parse copied to clipboard

This partially resolves iAmAsval/FModel#204, where we only have the "oldest" (closer to "biggest" in my investigation) asset available, even when a newer one is there. I dug into this and found the problem lies inside CUE4Parse. Currently CUE4Parse mounts paks parallelly and files in the last ready pak override the previous ones. That's the problem.

Two pieces of UE4 source code for reference:

1. FPakPlatformFile::GetPakOrderFromPakFilePath, basic order of paks in different folders. This isn't implemented in this PR.
2. FPakPlatformFile::Mount#L7239-L7265, order of numbered _P patches. This was not so detailed (UE<4.17) and I found some old articles complaining about it can't handle multiple patch paks well. Earlier games should have adapted their own ways (like, split into different parts) and I suppose that's the reason why iAmAsval/FModel#204 is raised only in recent months.

This PR

1. adds a ReadOrder property to IVfsReader to order readers. Implemented a simple _P pak ordering in PakFileReader.
2. adds a ReadOrder property to GameFile and adjusted it in VfsEntry to reflect the reader orderings. This is mainly to minimize changes to FileProviderDictionary, which receives path-GameFile pairs and I don't know if changing it would be too breaking.
3. adds a _byPath dictionary property to FileProviderDictionary, which only collects the GameFile of highest priority in each path.

I'm quite noob in C#, and want to have some practice here. If there's anything doesn't seem right, or u have some thoughts on the "basic order" above, let me know then I'll try my best. Or if you're planning a better approach, close this I'm willing to learn ;-)

This tool is amazing! Thank you!

It seems the reading order only matters on pak files, so I just reverted the changes to IVfsReader, GameFile, and VfsEntry. Now only PakFileReader and FileProviderDictionary are affected.

This seems to break the content changes to be reflected on the FileProviderDictionary after being passed to AddFiles.

Is it necessary? 👀 If it is, I will remove _byPath and instead check the priority in every TryGetValue call. If not, it seems better to require the newFiles passed to AddFiles to be a ReadOnlyDictionary instead of IReadOnlyDictionary, right?

Would these changes allow FModel to still show both copies, and read both separately?

@ryantheleach No yet. If I've read it right, FModel only passes the path to CUE4Parse, there's no way for CUE4Parse to know which copy FModel wants to load.

This PR only aims to make CUE4Parse provider to provide the file with the highest priority (as of now almost a random one is provided).

---

But I managed to made things work locally by combining with these changes:

1. exposing a static int PakFileReader.GetReadOrder(string pakName).
2. adding a new GameFile GetGameFile(string path, string? pakName = null) to AbstractFileProvider. When pakName is not null, it returns the GameFile with the largest order less than or equal to the order of pakName.
3. adding an optional parameter string? pakName = null to most members of AbstractFileProvider, use GetGameFile whenever applicable.
4. In FModel, CU4ParseViewModel.Extract, receive AssetItem asset instead of string fullPath, and then use asset.Package as the pakName passed to the members of Provider.

The problem is the speed. Whenever GetGameFile is called with a pakName string, the FileProviderDictionary would be iterate once. In my practice, it takes about extra 3 seconds when I load a material in a game with only 2 pak file.

---

Another way I can think of, instead of applying the steps above, would be to use multiple providers in FModel. That is, when a game has N paks, G.pak, G_0_P.pak, G_1_P.pak, ..., G_N-2_P.pak, generate N providers:

1. provider with G.pak,
2. provider with G.pak and G_0_P.pak,
3. provider with G.pak, G_0_P.pak, and G_1_P.pak, ...

And then in FModel we can read asset.Package to know which provider to use.

The problem now may be the memory. I'm not so familiar with CUE4Parse and FModel, I have no idea how this method can be applied and how the memory usage can be optimize.

---

Whichever way we choose, the prioritizing steps inside the provider (like this PR) is necessary.

Oh, the third method, integrating the second method into the GetGameFile in method 1. Whatever, the implementations are quite hard for me. It all depends on you guys. Just suggesting some possible ways. 👀

@ryantheleach Here in PaperStrike/FModel@extract-with-pak-name (with PaperStrike/CUE4Parse@pub-pak-order) has the first method implemented. 👀