bevyengine
Improve CompressedImageSaver
It's time to get serious about texture speed :). Streaming mips can come later, first we need a better system that actually compresses textures (on disk).
Old plan (ignore everything under this)
~~The current CompressedImageSaver outputs basisu files, which tend to be quite large, as they're designed for random-access decompression on the GPU, not for linear-reads like loading from disk. We should supercompress the textures by compressing the basisu with zstd. By storing that in a ktx2, we can reuse the existing ImageLoader (ktx2 zstd-supercompressed textures are an existing standard), no need for a custom loader.~~
~~We may also want to skip basisu, and just store BC7/ASTC/etc in supercompressed KTX2 files directly, to avoid the overhead, and let us support more formats. See the optional bullet points below. If so, a lot of the bullets below are invalid.~~
TODO
- Rename to CompressedTextureSaver, as it's only meant to be used for 3D textures
- Enable set_rdo_uastc()
- Might also need to use set_no_selector_rdo() for normal maps? See https://docs.rs/basis-universal/latest/basis_universal/encoding/struct.CompressorParams.html#method.tune_for_normal_maps
- After converting to basisu, encode to a zstd-supercompressed ktx2
- Add docs
- Explain disk compression vs GPU compressed texture formats
- Describe the pipeline: PNG/JPEG -> basisu -> zstd -> ktx2 -> BC7/ASTC
- Note that it generates mipmaps
- Explain why you want to do this: smaller size on disk, faster rendering
- Add an example (might also want to rework the asset_decompression example to work on something other than an image, or just replace it outright)
- Add support for the GLTF extension
KHR_texture_basisu(?)- Not sure if this is needed? Bevy can already support ktx2 files that store both basisu, and suprecompressed basisu + zstd data
- (Optional) Add support for saving to either BC7 and/or ASTC supercompressed ktx2 files directly (not sure how expensive the runtime transcoding from basisu is, might not matter)
- (Optional) Add support for BC6 textures and ASTC HDR for HDR content
- (Optional) Use BC5 for normal maps
Will this allow us to add mipmaps to a 2d image with multiple layers (for texture array) ?
Theoretically I think? Would probably need extra handling to deal with more than 1 layer. Depends how difficult that is to implement. I haven't figured out how to generate the mipmaps yet, might need to use wgpu and run some compute shaders.
After talking it over some more, I feel that basis universal is not worth it, and that we're better off using KTX2 with BC/ASTC textures already.
New thoughts:
- Still should be renamed to CompressedTextureSaver
- Still want more docs explaining it
- Pipeline is texture -> generate mips -> convert each mip to BC/ASTC -> zstd compress each mip -> write ktx2 file
Useful stuff:
- https://crates.io/crates/fast_image_resize for generating mips on the CPU, or GPU via compute shaders
- https://crates.io/crates/intel_tex_2 for BC/ASTC conversion
- According to a 2020 study https://aras-p.info/blog/2020/12/08/Texture-Compression-in-2020, bc7enc_rdo and astcenc give better results, but don't have maintained rust bindings. https://gpuopen.com/compressonator is also an option, and can do the whole pipeline for us.
- https://github.com/fintelia/ktx2encode and https://github.com/expenses/ktx2-tools/tree/master for writing ktx2 files
The core question around basisu is how to simultaneously support both desktop GPUs (which only support BCn) and non-Qualcomm mobile GPUs (which only support ASTC). There's a few possible approaches that I see:
- Encode to UASTC. This is basisu's higher quality format and takes 1 byte/pixel to store (or a bit less if you apply zstd afterwards).
- Create two ktx2 files with BC7 and ASTC compression respectively. Each file would be 1 byte/pixel for a total of 2 bytes/pixel (or a bit less if you apply zstd afterwards). Distribute different files depending on what the platform supports.
- Encode to BC7 and then transcode to ASTC at runtime (or vice versa) if the platform needs it. Only a single file with 1 byte/pixel, or a bit less with zstd. I haven't heard of anyone using this approach. The main downside is that on platforms that need transcoding you'd have slower loading and lower quality textures than either of the previous options.
- Is what I strongly suggest.
From what I've seen, basisu is just a lot slower, and not even worth it considering you're going to ship separate builds for mobile/desktop anyways. The only use case I see for it as web support, so I guess we could keep supporting it just for that.
Note the basis_universal crate doesn't support wasm: https://github.com/aclysma/basis-universal-rs/issues/10
There's also https://github.com/JakubValtar/basisu_rs which does pure-Rust UASTC transcoding, and thus should be wasm compatible
My current thinking is that we should just use https://developer.nvidia.com/gpu-accelerated-texture-compression.
There's also https://github.com/NVIDIA-RTX/RTXNTC, but it's not recommended yet for production use, and bevy is far from setup to use the neural textures directly (we would have to save as BC7).
Info dump:
Encoders: * ASTC: astc-encoder * BC4/5/7: bc7enc_rdo * BC6H: GPURealTimeBC6H * ETC2: etc2comp
Other interesting encoders: * https://github.com/hasenbanck/block_compression * https://developer.nvidia.com/gpu-accelerated-texture-compression * https://github.com/NVIDIA-RTX/RTXNTC
Preferred format: * Web - BasisU, or have the server send BC or ASTC based on client capabilities * macOS/iOS - ASTC * Android - ASTC, 80% coverage https://developer.android.com/guide/playcore/asset-delivery/texture-compression * Nvidia/AMD/Intel - BC * Really old devices - ETC2
BC subformats: BC7 - sRGB or sRGBA BC6H - HDR RGB BC5 - Normal map BC4 - Single channel data
Mipmaps - ?
3d textures - ?
File formats: Probably need a DDS/KTX -> KTX2 + zstd converter
I had considered that we should have solutions that cover (ultimately, but in no particular order):
- mipmap generation
- GPU-native formats (ASTC, BCn, ETC2, not UASTC nor ETC2S)
- high quality option for final export
- using the best quality encoders that operate in reasonable time
- probably the list of cli encoders @JMS55 mentioned above
- fast option for dev imports and iteration
- as M-series macs support BCn, this could be focused just on BCn as development is done on desktop-class GPUs
- something that works for importing textures on web
- perhaps the pure rust basisu encoder, or a non-Vida gpu-based one
And as a base requirement, I think whatever tools we choose must work across Windows/Linux/macOS, x86-64/arm64, and across AMD/Intel/NVIDIA/Apple M-series for doing the compression. I don’t mean one tool needs to be chosen that works everywhere, but rather that we need coverage. For example - choosing nvtt does not solve it for non-nvidia (right? Or at least not macOS, right?), so something else would be needed as well for non-nvidia/macOS.
nvtt can run CPU-only on windows and Linux. As there are Linux aarch64 binaries, it could be run in docker on macOS but that would be too awkward for most.
https://github.com/alecazam/kram Kram wraps bcenc, astcenc, and etc2comp, and it can output ktx2 and do mipmapping, has binaries/cam be built for windows/macOS/Linux. Perhaps leveraging that as a starting point for coverage of everything except web would be good?
The ASTC coverage seems low if we're going for vulkan capable android. https://vulkan.gpuinfo.org/listoptimaltilingformats.php?platform=android has it at ~99% supported.
