xiph
Skip applying high frequency DCT basis for larger size DCTs.
If the coefficients in high frequency subbands are not coded by PVQ (for lossy mode), then we can even skipping applying high frequency DCT basis. For example, for 64x64 DCT, if only top-left 32x32 DCT coefficients are encoded by PVQ, then during 2D DCT transform we can skip higher 32 frequency 1D basis in both horizontal and vertical 1D transform. However, if 2D DCT itself is implemented optimized way, skipping should be done accordingly. Btw, similar thing is true for decoder, which means if encoder never encode certain high freq subbands then, the iDCT in decoder don't need apply full basis function, but rather omitting high freq coefficients part of each basis will work since those are always multiplied to zeros.
This would only be true in keyframes: on delta frames the prediction may contain high frequencies, even if we do not code anything in those bands, and those frequencies need to survive a round-trip through the fDCT and iDCT.
Does this means that the predictor (i.e. from ref frame) should keep signals in all bands including high freq? So that original signal - predictor can be quantized and coded by PVQ?
We do not quantize and code "signal - predictor". We quantize and code "signal", using knowledge of "predictor". But when the quantized value for a band is just saying "use the predictor" (theta=0), we still need to feed the data from the predictor into the iDCT (and we had to get the data for those coefficients out of the fDCT).