Update to PackSquash v0.3.0

[AlexTMjugador]

Recently I've been putting some work on PackSquash, and as a result I've released the first (and hopefully last!) release candidate for the new v0.3.0 version, v0.3.0-rc.1. I wanted to do a release candidate before an actual release to gather some user feedback about any issues that may happen before a final release, buy myself a bit of time to do some last touches and write a proper changelog and more documentation, and give other projects, like Quiver and the brand new official GitHub action for PackSquash, the time they need to adapt to this new version, which isn't backwards compatible with v0.2.1 configuration files.

In particular, the options that v0.3.0 accepts have changed like this, with respect to v0.2.1:

New options:

• zip_compression_iterations
• work_around_minecraft_quirks
• size_increasing_zip_obfuscation
• percentage_of_zip_structures_tuned_for_obfuscation_discretion
• never_store_squash_times
• spooling_buffers_size
• open_files_limit
• zip_spec_conformance_level
• color_quantization_target
• minify_json
• delete_bloat_keys
• maximum_width_and_height
• minify_shader
• minify_properties

Removed options:

• strict_zip_spec_compliance (replaced by zip_spec_conformance_level)
• quantize_image (replaced by color_quantization_target)

Renamed options (no change in functionality):

• resource_pack_directory -> pack_directory
• compress_already_compressed_files -> recompress_compressed_files
• allowed_mods -> allow_mods

Of course, there are more details about how they work in the documentation.

This release is arguably the most ambitious PackSquash version ever, and I believe that it contains a lot of changes that may be useful for people that use Quiver, too. Some highlights about this release include:

• Support for identical file deduplication (i.e. if two files are identical after being processed, they can be stored only once in the generated ZIP file).
• ZIP files that PackSquash has generated in previous runs can now be used to vastly speed up next runs, because unchanged files will be copied over instead of being processed all over again. In order to be able to check for changes quickly, PackSquash relies on the file modification timestamps provided by the filesystem being reliable, though.
• Brand new and much more sophisticated extraction protection techniques, which resulted from my own original research on Minecraft internals and suggestions that people made over Discord.
• Fixed PackSquash not working for resource packs that contained audio files on macOS as much as possible.
• Support for comments in JSON files, and support for JSON files with .jsonc extension.
• Support for OptiFine Custom Entity Models files (.jem and .jpm).
• Support for data packs.
• Support for Opus audio files.
• Support for file names that contain non-ASCII characters and ZIP64 extensions, so now you can have more than 65536 files in a pack and reliably use more than 4 GiB in total. These files should be read fine by Minecraft.
• Substantially improved performance when dealing with big (> 64 MiB) packs, and in low available memory environments. The software architecture now supports dealing with files larger than the available memory, too.
• PackSquash is now much more robust: when lots of threads are used, it avoids exhausting the operating system open file limit; if an error happens, instead of risking generating incorrect output, it now stops.
• Text-based files (JSON, shaders, etc.) with a UTF-8 BOM are properly dealt with now, stripping the BOM during processing.
• Now key-value pairs that are known to be ignored by Minecraft in JSON files are removed by default. These pairs are added by some common model authoring tools, for example.
• Gigantic textures (> 4096x4096) are now rejected by default, because they usually make little sense to use with Minecraft. Limiting this ensures that PackSquash does not take too long processing them and helps authoring resource packs that work well accross a wide range of GPU maximum texture sizes.
• An option was added to customize the number of Zopfli compression iterations that are applied to files stored in the ZIP, to customize the desired tradeoff between compression and performance. Now the number of iterations done depends on the input file size, with smaller files being compressed more, and larger files being compressed less, so performance is more predictable and consistent.
• Fixed an issue that made grayscale-looking color images look wrong on older versions of Minecraft, by working around the underlying Minecraft quirk. This workaround must be explicitly enabled in the options file because it may hurt compression.
• Now shader files with Windows (CR + LF) line endings work with PackSquash. Previously they threw a parsing error.
• Audio files are no longer downmixed to mono by default, because this may surprise users ("why does this sound 3D now even though I did nothing?") and it did not yield significant space savings anyway, due to Ogg Vorbis joint encoding. It's still possible to downmix to mono or upmix to stereo using the options file, though, which is useful if you want to guarantee whether some sound will have 3D effects or not.
• Rewrite and refactor of the entire codebase. Now it is much more modular, in preparation for a future API. Automated unit tests were added. Lots of micro-optimizations were made, in regards to both memory usage and CPU usage. The application was migrated to an asynchronous programming model that leverages cooperative scheduling to improve thread utilization, which provides some speedups. In total, the total execution time was reduced a 5% in my tests, but YMMV depending on your pack (PNG file processing was optimized quite a bit by using faster codepaths, for example).
• Updated dependencies.
• Miscellaneous tidy-ups and improvements in the text strings that PackSquash outputs (now progress messages are logged to stderr, as other CLI programs do, like wget).
• Official builds for the Linux AArch64/ARM64 platform are now available (yes, you can run PackSquash on your Raspberry Pi now).

So, given all these changes, I think it'd be a nice thing to update. Let me know what you think!

[create-issue-branch[bot]]

Branch issue-107-Update_to_PackSquash_v0_3_0 created!