console
console copied to clipboard
sst
speed up autodeploy caching by ~10x by switching to zstd
this switches the compression algorithm for build caching from gzip to zstd, for approximately a ~10x speedup and noticeably better compression ratios. zstd is super well supported in the ecosystem and is already installed on the amazon linux OS the build containers are running. deploying this change should cause no customer disruption, i've switched the file extension to the standard .zst, so cache files previously stored via gzip will be ignored when restoring caches and ultimately cleaned up on the next force: true deploy.
the only customer-visible impact should be a cold cache on the first build after deploy, and then much faster caching afterward. rolling back, if necessary, should be similarly safe.
i looked into this because i cache node_modules in my monorepo, which is about ~2.5GB, and it was taking ~20 seconds to restore and ~1 minute to store. when running a version of this change on my own builds, restoring takes ~12 seconds and storing takes ~5-6 seconds. the compressed output also drops from ~450mb to ~350mb. this is consistent with publicly available benchmarks that show zstd is roughly 10x faster than gzip in normal cases.
i've wired up multicore compression as well, configured to scale with the size of the build machine up to a max of 8 cores. there are diminishing returns on spreading work across cores for zstd and using more cores is unlikely to improve speed (i confirmed this on my own build data). the bottleneck at this point is likely s3 bandwidth — not compression — which may have some future optimization potential.
