ArcticDB
ArcticDB copied to clipboard
Make all modification methods respect `prune_previous_versions`
Is your feature request related to a problem? Please describe.
Some methods in library.py
do not respect the prune_previous_versions
library option.
This makes the API inconsistent.
Discovered in #705
Describe the solution you'd like
- Make
sort_merge_internal
,sort_index
,delete_range_internal
anddefragment_symbol_data
C++ methods accept an extra argument and callwrite_version_and_prune_previous_if_needed
instead ofwrite_version
- Make the corresponding Python methods check the library option and pass it to C++. Also, consider adding a parameter to allow the user to override on each call.
- [x] Thoroughly check if there are other incorrect uses of
write_version
- [x] Consider if we need to bump the major version
Describe alternatives you've considered
- Leave the methods unchanged, which might lead to the original cause of #705 where a user unknowingly leave thousands of versions behind.
- Improve the documentation to ask the user to call
prune_previous_versions
after such methods. This is inefficient as the version chain might need reloading by then.
Some of these might already be fixed, we should check.
Step 1 of this should be a writeup with analysis of the current state of prune previous versions.
Step 2 tests reproducing the inconsistencies.
Step 3 fixes.
-
sort_merge
->sort_merge_internal
: prune_previous not available -
sort_index
: prune_previous not available -
delete_range
->delete_range_internal
: prune_previous not available -
defragment_symbol_data
: prune_previous not available