Trellis documentation lacking

[nagnug]

trafficstars

There needs to be explanation of the trellis feature, how it works, what else in the encoding process it interacts with. My interest is in quantization tables, so I want to know how the final matrix values are calculated with trellis enabled.

[fbossen]

The trellis feature essentially replaces the quantization block in the encoder. However the decoder is not modified and the quantization tables are not affected by the trellis feature. Give a quantization table the trellis process finds quantized transform coefficient values that minimize a rate-distortion metric of the form R + lambda*D. Note that when tuning for certain metrics such as PSNR and SSIM, a flat matrix is used instead of the default (JPEG Annex K) one.