chroma
chroma copied to clipboard
[BUG]: Deleting unloaded collection
Ref: #1309
Description of changes
Summarize the changes made by this PR.
- Improvements & Bug fixes
- Fixes an issue where collection data dir is only removed if the collection is first loaded into memory. We now use
hint_use_collection
to load the collection's segments before removal.
- Fixes an issue where collection data dir is only removed if the collection is first loaded into memory. We now use
Test plan
How are these changes tested?
- [x] Tests pass locally with
pytest
for python
Documentation Changes
N/A
Reviewer Checklist
Please leverage this checklist to ensure your code review is thorough before approving
Testing, Bugs, Errors, Logs, Documentation
- [ ] Can you think of any use case in which the code does not behave as intended? Have they been tested?
- [ ] Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
- [ ] If appropriate, are there adequate property based tests?
- [ ] If appropriate, are there adequate unit tests?
- [ ] Should any logging, debugging, tracing information be added or removed?
- [ ] Are error messages user-friendly?
- [ ] Have all documentation changes needed been made?
- [ ] Have all non-obvious changes been commented?
System Compatibility
- [ ] Are there any potential impacts on other parts of the system or backward compatibility?
- [ ] Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?
Quality
- [ ] Is this code of a unexpectedly high quality (Readbility, Modularity, Intuitiveness)
OMG, so many checks 😅
@Nayjest we take code quality very seriously! I think it has to be that way for shipping mission-critical software in open-source.
We could do this by changing local_persistent_hnsw to expose a delete that doesn't need to load the segment.
@HammadB, some random grpc server failure, can you rerun the test and if it fails again I'll keep digging.
Discussed offline: we can statically get_storage_folder and then delete if not loaded in the local segment_manager
@HammadB, I think this can be merged now.
Thanks! I hope that will help, coz I have now alomost terabyte DB on production that works crazily slow and actually stores less than 1KB of vectorized text ))
@HammadB since you approved, can this now be merged?
Can this issue be resolved please?