Brandon Liu

"Nested entries" for serialization in 5ec3a8188e2cfc5f9e5a23d88ac1c1f793ab457b this counterintuitively increased the total compressed index size from ~92 MB for a planet z14 basemap tileset to ~101 MB. Double checking if I...

No, because the signed varint encoding is zigzag-encoded, so they should still be small. The root cause here is another optimization which zeroes out offsets if the tile data is...

Nested method is 91.3 megabytes total on test dataset vs 92.8 megabytes non-nested

It shouldn't be hard, I will also try zstandard as well tonight, though I imagine the savings will be marginal for small files + high entropy numbers instead of strings

running brotli test in https://github.com/protomaps/PMTiles/tree/brotli

Note on the nested memory savings: With the existing non-nested implementation, the complete planet has about 40,000,000 entries, where each entry is 8 + 4 + 8 + 4 (tileid,runlength,offset,length)...

Brotli reduces the index size from 92.8 megabytes to 73.6 megabytes with the compression quality set to 11 (slowest), which is nice savings; it won't be practical until most runtimes...

zstd with the [default compression ratio (3)](https://github.com/klauspost/compress/tree/master/zstd#zstd) produces a 91.2 MB index, so almost the same as gzip; the higher compression levels are not implemented in this pure-Go port, and...

here's the real-world z14 test dataset I'm using to benchmark compression: https://protomaps-static.sfo3.digitaloceanspaces.com/dir.bin.gz The gzipped file is 293 MB, the ungzipped file is 981 MB. The file is simply a sequence...

@nyurik for most languages in practice I think it will not make a difference since the struct as a whole will need to be aligned to 8 bytes on 64...