DC trellis cost doesn't take into account contrast with previous line

[kornelski]

DC trellis cost is based only on difference of current block's desired vs quantized value. This doesn't take into account a false edge quantization could create.

For example if first row's quantization error is -1, and in a second row's the error is +1, then the difference between these rows is almost as noticeable as quantization error +2 would create alone.

In other words, if a previous row was darkened, then there's an extra cost to lightening the next row, as it will increase contrast between the rows and create a more visible edge.

Here's an extreme example that shows vertical banding:

In quantize_trellis() I suggest adding a fraction of previous row's quantization error to the delta used for dc_candidate_dist.

[bdaehlie]

What do people think about blocking 3.0 on this? If we don't block it doesn't mean we can't fix this, it just means it'll be in a later release.

If we want to block 3.0 on this, someone needs to sign up to fix within the next week.

[kornelski]

I think it can wait. It's more of a feature than a bug. It's visible only in low quality.