Improve distance metric in block size RDO

[jmvalin]

The current one is a quick hack, I can probably get a lot better, e.g. a metric similar to the old switching code, but using the real rate and quantized data.