quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

We should run one GC actor per index, not per source.

Open fulmicoton opened this issue 3 years ago • 2 comments

We run one GC per source, but should run one GC per index

9876568054679316988002] shardId-000000000007:(..49630019518881077165006813008667182781726868974027669618]) checkpoint=Ckpt()
2022-05-31T09:20:38.735Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=gh-archive gen=0}:{actor=GarbageCollector}:{msg_id=1900}: quickwit_indexing::actors::garbage_collector: garbage-collect-operation
2022-05-31T09:20:38.735Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=gh-archive gen=0}:{actor=GarbageCollector}:{msg_id=1900}: quickwit_indexing::actors::garbage_collector: garbage-collect-operation
2022-05-31T09:21:38.737Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=gh-archive gen=0}:{actor=GarbageCollector}:{msg_id=2000}: quickwit_indexing::actors::garbage_collector: garbage-collect-operation
2022-05-31T09:21:38.737Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=gh-archive gen=0}:{actor=GarbageCollector}:{msg_id=2000}: quickwit_indexing::actors::garbage_collector: garbage-collect-operation
2022-05-31T09:22:38.738Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=gh-archive gen=0}:{actor=GarbageCollector}:{msg_id=2100}: quickwit_indexing::actors::garbage_collector: garbage-collect-operation
2022-05-31T09:22:38.738Z  INFO {actor=quickwit_indexing::actors::indexing_service::IndexingService}:{msg_id=1}::{index=gh-archive gen=0}:{actor=GarbageCollector}:{msg_id=2100}: quickwit_indexing::actors::garbage_collector: garbage-collect-operation
``

fulmicoton avatar May 31 '22 09:05 fulmicoton

@fulmicoton

  • Pulling the GC actor from the indexing_pipeline, I end up with an unsupervised actor. is this ok? We probably have to log its errors to keep it running like the IngestAPI queue GC
  • We also need to shutdown the GC actor when the last index_pipeline for this index is shutdown, right?

evanxg852000 avatar Jun 07 '22 12:06 evanxg852000

It is slightly more involve. Let's not address this ticket in 0.3.1. It is related to #1613.

Assigning it temporarily to myself, and will reassign it when it is ready.

fulmicoton avatar Jun 08 '22 09:06 fulmicoton

This issue is fixed. The GC has moved to the janitor service.

guilload avatar Oct 09 '22 22:10 guilload