foundationdb-dotnet-client icon indicating copy to clipboard operation
foundationdb-dotnet-client copied to clipboard

Transaction Size does not take into account read and write conflict ranges

Open KrzysFR opened this issue 7 years ago • 1 comments

Looking at the code in fdbclient (Transaction::commitMutations()) it looks like starting from API version 300+, the total transaction size is the sum of the mutated key/value (writes), as well as the size of all the read and write conflict ranges.

A write with key "FOO" and value '"Hello World"` should add the following overhead:

  • Mutations: (Set, 'FOO', 'Hello World').expectedSize() = 3 + 11 = 14 bytes
  • Read conflicts: None
  • Write conflicts: ('FOO', 'FOO\0') = 3 + 4 = 7 bytes

The total footprint of this operation would be 21 bytes instead of 14 bytes as is currently guesstimated by the .NET binding. This means that algorithms that try to batch data by following the transaction size are underestimating the actual size, and could fail with transaction_too_large while thinking this is ok.

KrzysFR avatar May 12 '18 14:05 KrzysFR

There is a new API in fdb 6.2 that allows to get a estimate of the transaction size, which should be more precise than the current heuristic.

Only issue is that the new API is async, which is a breaking change, and also may change the perf characteristics of existing implementations.

KrzysFR avatar Nov 27 '19 14:11 KrzysFR