Encryption: Change the nonce during encryption

[nikipapadatou]

We expected that encryption by default would improve the batch utilisation, but after we tried this, it seems this is not the case. The idea has been that batch utilisation would improve because they would be evenly distributed in the buckets. However, in the current implementation when we encrypt chunks, we use a sequential counter. This means that we are using the same random XOR nonce every time to randomise them and then distribute them randomly. So, according to the birthday paradox, the reason our tries had no effect was because we would have the same level of randomness (same nonce) for all the chunks during encryption - in simple words, we had exactly the same randomness as before.

We need random data that is uniformly distributed to the buckets for this to work. In order to achieve this, we need a different nonce during encryption so that they are evenly distributed. Instead of the current sequential counter, we need an arbitrary counter that isn't repeated or reused and the nonce need to be mined in the encryption - we need to change our current random encrypt function to increment the nonce. What we want to achieve is having encrypted data uniformly distributed into the buckets and this will be achieved by altering the nonce every time with one that has not been used before, we will get encrypted random data.

Given the above, we expect that after this change, encryption by default will bring us the expected results with improving the batch utilisation.

[nikipapadatou]

Let's sync in the future with the research team to see how relevant or even accurate this is. There have been thoughts that probably this would not make any difference at all to the batch utilisation, nor would change the randomness or so. Let's discuss it again in the future.