[BUG]: Deleting unloaded collection

[tazarov]

Ref: #1309

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • Fixes an issue where collection data dir is only removed if the collection is first loaded into memory. We now use hint_use_collection to load the collection's segments before removal.

Test plan

How are these changes tested?

• [x] Tests pass locally with pytest for python

Documentation Changes

N/A

[github-actions[bot]]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• [ ] Can you think of any use case in which the code does not behave as intended? Have they been tested?
• [ ] Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• [ ] If appropriate, are there adequate property based tests?
• [ ] If appropriate, are there adequate unit tests?
• [ ] Should any logging, debugging, tracing information be added or removed?
• [ ] Are error messages user-friendly?
• [ ] Have all documentation changes needed been made?
• [ ] Have all non-obvious changes been commented?

System Compatibility

• [ ] Are there any potential impacts on other parts of the system or backward compatibility?
• [ ] Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• [ ] Is this code of a unexpectedly high quality (Readbility, Modularity, Intuitiveness)

[Nayjest]

OMG, so many checks 😅

[jeffchuber]

@Nayjest we take code quality very seriously! I think it has to be that way for shipping mission-critical software in open-source.

[HammadB]

We could do this by changing local_persistent_hnsw to expose a delete that doesn't need to load the segment.

[tazarov]

@HammadB, some random grpc server failure, can you rerun the test and if it fails again I'll keep digging.

[HammadB]

Discussed offline: we can statically get_storage_folder and then delete if not loaded in the local segment_manager

[tazarov]

@HammadB, I think this can be merged now.

[Nayjest]

Thanks! I hope that will help, coz I have now alomost terabyte DB on production that works crazily slow and actually stores less than 1KB of vectorized text ))

[prevosti]

@HammadB since you approved, can this now be merged?

[rv3183]

Can this issue be resolved please?

[jcorrie]

I've just experienced this bug. @HammadB - is the plan still to merge this PR?