quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

Garbage collect ingested files in the metastore in Lambda indexer

Open fmassot opened this issue 6 months ago • 0 comments

Each time we invoke the lambda indexer to ingest a file, it will index it and store a new checkpoint with the file's name.

The metastore file will become bigger and bigger. We can remove the checkpoints after indexing and leave the responsibility to the caller to not index the same file twice. It could also be a parameter that we can pass to the indexer.

fmassot avatar Feb 20 '24 10:02 fmassot