Optimize DepotManifest Serialization

[skylayer]

This PR builds on PR #1113, introducing a HashSet with ChunkIdComparer to manage unique chunks in DepotManifest serialization, significantly reducing processing time.

My comment

Changes:

• Replaced List<byte[]> with HashSet<byte[]> for unique chunks.
• Added ChunkIdComparer for efficient byte array comparison.

Impact:

• Before: Serialization took up to 10 minutes.
• After: Reduced to ~0.61 seconds.

This optimization streamlines serialization, especially for large manifests, enhancing performance without data integrity loss.

[xPaw]

Why does it even need to check for duplicates?

[skylayer]

I'm not entirely sure why the original author included the duplicate checks—probably had a good reason. I just worked on speeding up the process they set up. 🤷‍♂️

[skylayer]

BTW, I'm thinking of making the Serialize() function in DepotManifest public. Right now, we've only got SaveToFile accessible, which kinda boxes us in. If Serialize() were public, we could directly handle more scenarios, like different formats or custom storage solutions.

[NicknineTheEagle]

Why does it even need to check for duplicates?

Because Valve's manifest format stores the number of unique chunks, I just followed it.

[NicknineTheEagle]

BTW, I'm thinking of making the Serialize() function in DepotManifest public. Right now, we've only got SaveToFile accessible, which kinda boxes us in. If Serialize() were public, we could directly handle more scenarios, like different formats or custom storage solutions.

I checked the original PR and I don't know why I didn't make Serialize public, to be honest.

[yaakov-h]

btw congrats on getting PR number 1337 😉