PCSX2
[BUG]: Dumping Creates Duplicate Textures With Non-Unique Name Hashes
Describe the Bug
Several games when dumping textures will produce tens of thousands of textures in a few seconds of dumping. Ace Combat 4, 5, and Zero are known culprits. In these cases there are dozens of identical files for each unique texture. When the dumped PNG files are checked for similarity they are identical in every way except for name. There appears to be something wrong with the hashing algorithm that generates different names for identical images.
For example the following images are identical but the first portion of the image name hash differs.
1eb7cc3b1d5fa43d-af099a789bab7043-r32x32-00002213.png
2dfec3fde93735b1-af099a789bab7043-r32x32-00002213.png
2e6cf21fc0212e03-af099a789bab7043-r32x32-00002213.png
3e9ed7fc7d1e9b4e-af099a789bab7043-r32x32-00002213.png
4f7190f881ae0940-af099a789bab7043-r32x32-00002213.png
Reproduction Steps
Run a mission in Ace Combat 4, 5, or Zero and turn on texture dumping. Thousands of ground textures will fill the dump folder.
Expected Behavior
Identical images in graphics memory should have a single unique hash.
PCSX2 Revision
v2.3.246
Operating System
Windows 11
If Linux - Specify Distro
No response
Logs & Dumps
No response
You're not alone.
307,979 textures.....
Look at the grey weird gradient textures that is all over. If I turn around and do a texture replacement with these generated textures it is for sure very wrong.
This is using Nightly builds and I've been playing the game I've wanted to collect textures for on and off over the past 2 months if I had to guess a time frame.
@shuttle128 @Chromos-Def If you run the seemingly duplicate images through a diff checker, are they all identical? Can you also verify all the duplicates have the same CLUTs?
@shuttle128 @Chromos-Def If you run the seemingly duplicate images through a diff checker, are they all identical? Can you also verify all the duplicates have the same CLUTs?
Checking if they have the same checksum but different file name?
Also how do I check on their CLUTS?
Checking if they have the same checksum but different file name?
Yes, specifically the first two components of the file name.
Also how do I check on their CLUTS?
The CLUT is one of the components of the file name, I think the last one. Point being, check if "duplicates" are the same file name but a different CLUT at the end.
The reason I ask is because texture dumping on the PS2 is extremely literal minded and is done by just dumping any texture as the game uploads it to the GS. There is no regard for how much or how little the game actually sanitized the memory being uploaded; it is semi-regular that a game uploads a texture to the GS but the memory allocated for the texture still contains old or uninitialized memory which is basically going to change every frame as that memory gets used for other things and the textures are constantly being allocated memory and invalidated.
Indeed, size changes would indicate the different mipmap levels for a single texture as well.
So I tried to whip something up
Base Hash: 36c8d5d453202d84 (2 variants)
36c8d5d453202d84-d7f477be4e22155b-00001553.png 36c8d5d453202d84-dbdc863b62daba61-00001553.png
Base Hash: 4116225e0456464d (4 variants)
4116225e0456464d-58fe77f450a0131d-00002213.png 4116225e0456464d-5b39987b13d18cab-00002213.png 4116225e0456464d-be73d89bef3fc557-00002213.png 4116225e0456464d-e5fca7b88cbe8074-00002213.png
Base Hash: 6ea862afabd0f36e (4 variants)
6ea862afabd0f36e-2480bdae2824d5c9-00001553.png 6ea862afabd0f36e-55279158ecd51bed-00001553.png 6ea862afabd0f36e-a8f3383965766bdd-00001553.png 6ea862afabd0f36e-c13e7319bfd57f76-00001553.png
Base Hash: 89019587fe0fa821 (2 variants)
89019587fe0fa821-cc91661d1baa53fe-00001553.png 89019587fe0fa821-e4a2a60faacf24db-00001553.png
Base Hash: a68dfbb1d75c6e8d (3 variants)
a68dfbb1d75c6e8d-391cc5c3c9ab00cb-00001113.png a68dfbb1d75c6e8d-58e483cd9b60b8ac-00001114.png a68dfbb1d75c6e8d-b9edc87d40f34df3-00001113.png
Base Hash: cd953d142bdb2d1f (2 variants)
cd953d142bdb2d1f-f88a7cd21e8c2d06-00001553.png cd953d142bdb2d1f-ff294cb15e31e96a-00001553.png
Base Hash: e72b8d33114bf67f (2 variants)
e72b8d33114bf67f-3ad9d97f4c8df617-00001553.png e72b8d33114bf67f-a4ec5128c6af4e0a-00001553.png
Does this make sense?
I couldn't find anything that had the same first and second hash names.
The two components are different parts of the texture. The first is the base data, I don't recall what the second part is. The fact that the second component is different means these are 2+ distinct textures each.
I meant check the MD5 of the literal files which appear to be duplicates, and see if they are the same MD5. That will tell you if the raw texture data is identical or not. My hunch is the MD5 sums of what appear to be duplicates will in fact be different, meaning you are running into the classic problem that all other games exhibiting this behavior do, where they upload textures including some amount of uninitialized data and generating an infinite pile of textures.
Ok the checksum finally got done running xD 307,979 files took a hot minute to check.
Here's a small snippet of the results
Found 62609 duplicate sets
Checksum: 13e6690060e2ca3f219723faa35940da643382e9fdc46cbbf18557d45ffe9562 (3 occurrences)
------------------------------------------------------------
C:\Games\PCSX2-Nighty\textures\SLUS-20149\dumps\100038efc4ea07b6-6d9168b3e06cb164-00001993.png
C:\Games\PCSX2-Nighty\textures\SLUS-20149\dumps\87b71959a3d86ef0-6d9168b3e06cb164-00001993.png
C:\Games\PCSX2-Nighty\textures\SLUS-20149\dumps\b64f25556935baa4-6d9168b3e06cb164-00001993.png
Checksum: 61da2046f7c65943bc3b511951bb3c9f3bfeabc82c76178f11da4cf17bffe6c7 (2 occurrences)
------------------------------------------------------------
C:\Games\PCSX2-Nighty\textures\SLUS-20149\dumps\10004bb8ed3e41e8-6d9168b3e06cb164-00001993.png
C:\Games\PCSX2-Nighty\textures\SLUS-20149\dumps\d49366d03eca9629-6d9168b3e06cb164-00001993.png
Checksum: 36ab439a3dc70dd7cac1fe766620a5d557d0dfaf41161c53272711df676a356f (2 occurrences)
------------------------------------------------------------
C:\Games\PCSX2-Nighty\textures\SLUS-20149\dumps\100063a842e74c2d-6d9168b3e06cb164-00001993.png
C:\Games\PCSX2-Nighty\textures\SLUS-20149\dumps\f228d3b3b1ff5c9d-6d9168b3e06cb164-00001993.png
Checksum: 536f48378384c595646e728e8ce4d0fdba519f826f53b0dd96b509712ed8e697 (4 occurrences)
------------------------------------------------------------
C:\Games\PCSX2-Nighty\textures\SLUS-20149\dumps\100108ed0586cdd9-6d9168b3e06cb164-00001993.png
C:\Games\PCSX2-Nighty\textures\SLUS-20149\dumps\1d39cf70738a31bc-6d9168b3e06cb164-00001993.png
C:\Games\PCSX2-Nighty\textures\SLUS-20149\dumps\5bfbea12840c1b89-6d9168b3e06cb164-00001993.png
C:\Games\PCSX2-Nighty\textures\SLUS-20149\dumps\de68560ce29b1899-6d9168b3e06cb164-00001993.png
If you'd like I can upload the full .txt file somewhere since it is massive with 62609 duplicate sets.
Mildly interesting, the base hash is in fact changing despite the same raw texture data. There may be more to it that is beyond me (there could be other information in the first hash component which isn't represented in the image data), but this does suggest something is happening with the hash changing. Someone smarter than me would have to comment on this.
I appreciate you diving into this with me. Hopefully the information that was mustered up can be of some use to someone more keen on this issue, if the right eyes happen to stumble upon this issue.
@shuttle128 @Chromos-Def If you run the seemingly duplicate images through a diff checker, are they all identical? Can you also verify all the duplicates have the same CLUTs?
I checked the resulting PNGs in a hex editor for differences and they are all identical. As you can see from the PCSX2 generated file names for all of the ones I posted, the final number is identical which should indicate the CLUTs are all the same.
I am aware of the issue with the GS not always garbage collecting and how some textures are different because of leftovers in memory making the textures non-identical. These examples do not appear to be this issue. These are identical images.
I have also checked the MD5 of the images I uploaded. Results are below:
I attempted to go through the PCSX2 source to see if I could untangle exactly how the hash is built for the dumped files. It's been a long time since I looked at anything C++ but it seemed like CreateTextureName in GSTextureReplacements.cpp would be the place to look. To me it looks like what goes into the hashes are pixel storage mode, texture width and height, alpha TA0, alpha enable mask, alpha TA1, texture bitmap, CLUT, mip level, then region width and height. I don't have enough experience with C++ to understand how that results in the actual filename, but maybe that helps to narrow it down.
In Ace Combat 5 it appears that the issue only affects the 32x32 and 16x16 mip map textures though. The 64x64 textures seem to be pretty stable (the highest resolution mip map for ground textures).
Here are another 11 files with the same issue:
A similar issue is known to affect EA Sports NASCAR games (Thunder 2002 to NASCAR 09) with caveat that dumps produced for those games have different garbage data surrounding them.
For instance, these three are essentially the same texture (Dale Earnhardt Jr.'s 2003 main paint scheme/livery in NASCAR Thunder 2004):
e5ba70416987e472-7564936e532bdda0-00006653.png (MD5: 7af69c3fb49c4ecfbc215dd5b2433575) - consider this the "main" one as it was the first dumped.
31f67277fe68372-7564936e532bdda0-00006653.png (MD5: 9f58eecd465232f886c4103dcb4a2475)
eb4dfdf6acf93e76-7564936e532bdda0-00006653.png (MD5: 3dc7784bbb24c0e5edf5573271dfbec6)
A similar issue is known to affect EA Sports NASCAR games (Thunder 2002 to NASCAR 09) with caveat that dumps produced for those games have different garbage data surrounding them.
This is a very well known, unrelated and unsolvable problem. If the game developers did not either clear the VRAM allocated for the texture before using it, and/or did not set the boundaries of the texture to the portion they were actually using, then this will happen and there is nothing that can fix it other than if someone were to rewrite the game to not do this behavior.
Edit: FWIW this is exactly what I initially chalked up the original report of this issue to. It wasn't until talking through it with the original posters that I could see what they were talking about - the literal texture data in their examples is a perfect match in each dump with matching MD5 sums and all. And yet theirs are dumping multiple times with the first component of the file name being different, in which case we would expect the literal texture data to be different too.
What remains to be seen for the original report is if this is some more niche detail that I don't know enough about. If the first component of the hash is not just for the raw texture data but also includes color or mip or some other metadata, then that would indicate this behavior in OP is correct. If not, then that means something is going on here. But I'm not qualified to say if that's the case or not. We'll have to wait until someone who understands the texture dumping system better can speak on the exact specifics of the hash components.
Same issue with duplicate textures (and many more, including broken textures) exists in Indiana Jones and the Emperor's Tomb:
28b81fff5a79d847-c19fd9a464dd6a86-00001993.png
30dbdeb3bc350406-e458887e9b47641d-00001993.png
50ebffd2ebdc2cc2-222e096eb6d04e09-00001993.png
62c73cd4894a04b4-54933023414e3fc3-00001993.png
43aef851c82c31bb-9446129e51b86230-00002214.png
56e9f10005312ce9-9446129e51b86230-00001dd4.png
Could you please fix the texture dumping in that game? The Emperor's Tomb has one of the worst textures in the entire PS2 game library (they are so bad that it looks like a PS1 game) and is in a dire need of an HD texture pack, but creating a proper one, based on PC or Xbox textures, is very challenging because the PCSX2 texture dumping doesn't work well in that game as of now.
The same thing with Harry Potter and the Chamber of Secrets: 10,000 corrupted and duplicate textures were dumped in just the first two minutes of gameplay, with only a handful of textures that can be upscaled:
Same issue with duplicate textures (and many more, including broken textures) exists in Indiana Jones and the Emperor's Tomb:
28b81fff5a79d847-c19fd9a464dd6a86-00001993.png
30dbdeb3bc350406-e458887e9b47641d-00001993.png
50ebffd2ebdc2cc2-222e096eb6d04e09-00001993.png
62c73cd4894a04b4-54933023414e3fc3-00001993.png
43aef851c82c31bb-9446129e51b86230-00002214.png
56e9f10005312ce9-9446129e51b86230-00001dd4.png
Could you please fix the texture dumping in that game? The Emperor's Tomb has one of the worst textures in the entire PS2 game library (they are so bad that it looks like a PS1 game) and is in a dire need of an HD texture pack, but creating a proper one, based on PC or Xbox textures, is very challenging because the PCSX2 texture dumping doesn't work well in that game as of now.
This case does not appear to be related to this issue. Those images do not appear to be identical. This issue is only about PCSX2 dumping many identical images with non-identical names. Each instance of a texture should have a unique hash/name. All six of your textures are obviously visually different.
For reference, Dolphin fixed a similar issue back in the day:
- https://github.com/dolphin-emu/dolphin/pull/1885
- https://github.com/dolphin-emu/dolphin/commit/d27bd9d291a178892674cdffc397b586b37c0f2b
