RSPTCPServer icon indicating copy to clipboard operation
RSPTCPServer copied to clipboard

8bit quantization

Open CornN64 opened this issue 3 years ago • 0 comments

To improve the 8bit quantization in the RTL mode with ½ LSB and round it to the nearest LSB I would suggest to replace the lines

*(data++) = (unsigned char)(((*xi << sample_shift) >> 8) + 128); *(data++) = (unsigned char)(((*xq << sample_shift) >> 8) + 128);

with

*(data++) = (unsigned char)((((*xi << sample_shift) + 0x80) >> 8) + 128); *(data++) = (unsigned char)((((*xq << sample_shift) + 0x80) >> 8) + 128);

or even more optimized

*(data++) = (unsigned char)(((*xi << sample_shift) + 0x8080) >> 8); *(data++) = (unsigned char)(((*xq << sample_shift) + 0x8080) >> 8);

Cheers

CornN64 avatar Jul 01 '21 15:07 CornN64