quickwit
quickwit copied to clipboard
Garbage collect ingested files in the metastore in Lambda indexer
Each time we invoke the lambda indexer to ingest a file, it will index it and store a new checkpoint with the file's name.
The metastore file will become bigger and bigger. We can remove the checkpoints after indexing and leave the responsibility to the caller to not index the same file twice. It could also be a parameter that we can pass to the indexer.