weaviate icon indicating copy to clipboard operation
weaviate copied to clipboard

Published 20 hours ago • verified publisher icon weaviate

Disk (or mmap) based HNSW commit log condensor

Open etiennedi opened this issue 3 years ago • 11 comments

Background

The current condensor loads an uncondensed commit log into memory and then writes the condensed version back to disk. This creates two problems:

  1. Either we have massive memory spikes that would become very costly in larger datasets or we need to cap the max size of what can be condensed to something reasonable
  2. With the max size being capped, over time we won't be able to condense logs anymore and a) we need too much disk space and b) we vastly increase the startup times of large datasets potentially impacting MTTR, etc.

cc @cdpierse

Suggestion / Idea

A disk based condensor that either holds nothing in memory or just very simple index structures, so that the actual payload is written straight to disk. This way memory spikes would be minimal and we could tackle basically any size, eliminating the problems mentioned above.

etiennedi avatar Aug 13 '21 09:08 etiennedi

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Oct 12 '21 14:10 stale[bot]

Unmark stale.

etiennedi avatar Oct 12 '21 15:10 etiennedi

Unmark stale.

Thanks for ignoring this, stale not!

etiennedi avatar Oct 20 '21 19:10 etiennedi

Thank you for your contribution to Weaviate. This issue has not received any activity in a while and has therefore been marked as stale. Stale issues will eventually be autoclosed. This does not mean that we are ruling out to work on this issue, but it most likely has not been prioritized high enough in the last months. If you believe that this issue should remain open, please leave a short reply. This lets us know that the issue is not abandoned and acts as a reminder for our team to consider prioritizing this again. Please also consider if you can make a contribution to help with the solution of this issue. If you are willing to contribute, but don't know where to start, please leave a quick message and we'll try to help you. Thank you, The Weaviate Team

stale[bot] avatar Dec 22 '21 11:12 stale[bot]

Keep

etiennedi avatar Dec 22 '21 11:12 etiennedi

Reopen

etiennedi avatar Dec 29 '21 14:12 etiennedi

Thank you for your contribution to Weaviate. This issue has not received any activity in a while and has therefore been marked as stale. Stale issues will eventually be autoclosed. This does not mean that we are ruling out to work on this issue, but it most likely has not been prioritized high enough in the last months. If you believe that this issue should remain open, please leave a short reply. This lets us know that the issue is not abandoned and acts as a reminder for our team to consider prioritizing this again. Please also consider if you can make a contribution to help with the solution of this issue. If you are willing to contribute, but don't know where to start, please leave a quick message and we'll try to help you. Thank you, The Weaviate Team

stale[bot] avatar Apr 24 '22 04:04 stale[bot]

Keep

etiennedi avatar Apr 24 '22 09:04 etiennedi

Thank you for your contribution to Weaviate. This issue has not received any activity in a while and has therefore been marked as stale. Stale issues will eventually be autoclosed. This does not mean that we are ruling out to work on this issue, but it most likely has not been prioritized high enough in the last months. If you believe that this issue should remain open, please leave a short reply. This lets us know that the issue is not abandoned and acts as a reminder for our team to consider prioritizing this again. Please also consider if you can make a contribution to help with the solution of this issue. If you are willing to contribute, but don't know where to start, please leave a quick message and we'll try to help you. Thank you, The Weaviate Team

stale[bot] avatar Jun 29 '22 12:06 stale[bot]

Keep

etiennedi avatar Jul 01 '22 11:07 etiennedi

Stale bot is really trying to get rid of this one 💨

parkerduckworth avatar Aug 09 '22 00:08 parkerduckworth

Thank you for your contribution to Weaviate. This issue has not received any activity in a while and has therefore been marked as stale. Stale issues will eventually be autoclosed. This does not mean that we are ruling out to work on this issue, but it most likely has not been prioritized high enough in the last months. If you believe that this issue should remain open, please leave a short reply. This lets us know that the issue is not abandoned and acts as a reminder for our team to consider prioritizing this again. Please also consider if you can make a contribution to help with the solution of this issue. If you are willing to contribute, but don't know where to start, please leave a quick message and we'll try to help you. Thank you, The Weaviate Team

stale[bot] avatar Oct 15 '22 22:10 stale[bot]