allwpilib
allwpilib copied to clipboard
Add compression option for WPILOGs
Many teams are using the WPILOG format to store hundreds or thousands of fields, and often without removing duplicate values. This can result in enormous log files that are difficult to download and can quickly fill the RIO's storage. Adding an option to compress data before writing to disk would alleviate this issue, and could remain optional by adding a compression flag to the WPILOG header.
Of course, choosing an algorithm and configuration that can perform well on the RIO is an important consideration. However, we already have evidence that this is reasonable given CTRE's (apparent) use of compression for the Hoot log format without issue.
I ran a couple of tests using zlib (with the same configuration as gzip) to determine how effective compression might be on a typical WPILOG. This is based on a ~4.5 minute log file of a real match with 533 fields, including a variety of data types (arrays, structs, simple values, etc).
- Without duplicate values: 23.3 MB compresses to 11.5 MB (49.4%)
- Including duplicate values: 178.1 MB compresses to 31.7 MB (17.8%)