Jump in CSUP file size

[philrz]

It's been noted that the size of a CSUP file generated from the GitHub archive data used in the super command perf queries jumped significantly due to the changes in #5569. While this may just be a necessary cost to get the other benefits of that change, the team plans to improve the CSUP writer so there may be some improvements in file size that come out of that effort.

Details

Here's the 3.5 GB file size at commit 3f5ae2e right before the changes landed from #5569:

$ super -version
Version: v1.18.0-220-g3f5ae2e47

$ super -f csup -o gha-3f5ae2e.csup gharchive_gz/*

$ du -sh gha-3f5ae2e.csup
3.5G	gha-3f5ae2e.csup

Then after the changes from #5569 (commit 5658bc5) it jumps to 6.1 GB:

$ super -version
Version: v1.18.0-221-g5658bc586

$ super -f csup -o gha-5658bc5.csup gharchive_gz/*

$ du -sh gha-5658bc5.csup
6.1G	gha-5658bc5.csup

While there's been plenty of CSUP work in the months since, the CSUP size remains roughly the same at tip of main (currently commit 22bf163).

$ super -version
Version: v1.18.0-388-g22bf16383

$ super -f csup -o gha-22bf163.csup gharchive_gz/*

$ du -sh gha-22bf163.csup
6.2G	gha-22bf163.csup

By comparison, BSUP is less than half the size.

$ super -version
Version: v1.18.0-388-g22bf16383

$ super -f bsup -o gha-22bf163.bsup gharchive_gz/*

$ du -sh gha-22bf163.bsup
2.8G	gha-22bf163.bsup

FWIW, the original data is 2.5 GB of GZIP-compressed JSON which is 17.1 GB uncompressed.

$ du -sh gharchive_gz/
2.5G	gharchive_gz/

$ gzcat gharchive_gz/* | wc -c
 18396198934