entity-fishing icon indicating copy to clipboard operation
entity-fishing copied to clipboard

Add a normalization after embeddings quantization

Open lfoppiano opened this issue 6 years ago • 1 comments

The quantisation of the embeddings (#46) wasn't correcting mapping the weight on values from -127 to +127, so it was choosen to store them into shorts instead of bytes.

With byte storage we would save more disk. It's anyway good idea to do a general review to understand why we have such problem with weight quantisation.

lfoppiano avatar Mar 17 '18 18:03 lfoppiano

Quantization has no problem and is doing what is expected, there is requirement of having everything in the interval [-128, 127], what was not correct is that I thought everything was in this range and we could use byte for storing quantized embeddings.

What is needed is a projection on this range of the actual quantized embeddings as additional step, but I am not sure which distribution function to use.

I update the title of the issue to reflect this.

kermitt2 avatar Mar 17 '18 18:03 kermitt2