ArcticDB icon indicating copy to clipboard operation
ArcticDB copied to clipboard

Make all modification methods respect `prune_previous_versions`

Open qc00 opened this issue 1 year ago • 2 comments

Is your feature request related to a problem? Please describe.

Some methods in library.py do not respect the prune_previous_versions library option. This makes the API inconsistent.

Discovered in #705

Describe the solution you'd like

  • Make sort_merge_internal, sort_index, delete_range_internal and defragment_symbol_data C++ methods accept an extra argument and call write_version_and_prune_previous_if_needed instead of write_version
  • Make the corresponding Python methods check the library option and pass it to C++. Also, consider adding a parameter to allow the user to override on each call.
  • [x] Thoroughly check if there are other incorrect uses of write_version
  • [x] Consider if we need to bump the major version

Describe alternatives you've considered

  • Leave the methods unchanged, which might lead to the original cause of #705 where a user unknowingly leave thousands of versions behind.
  • Improve the documentation to ask the user to call prune_previous_versions after such methods. This is inefficient as the version chain might need reloading by then.

qc00 avatar Aug 10 '23 08:08 qc00

Some of these might already be fixed, we should check.

Step 1 of this should be a writeup with analysis of the current state of prune previous versions.

Step 2 tests reproducing the inconsistencies.

Step 3 fixes.

poodlewars avatar Apr 18 '24 12:04 poodlewars

  • sort_merge -> sort_merge_internal: prune_previous not available
  • sort_index: prune_previous not available
  • delete_range -> delete_range_internal: prune_previous not available
  • defragment_symbol_data: prune_previous not available

phoebusm avatar May 03 '24 16:05 phoebusm