quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

Implement `reset_index_checkpoint` and `reset_source_checkpoint` metastore APIs and CLI commands

Open guilload opened this issue 2 years ago • 5 comments

Description

Per title.

How was this PR tested?

  • added unit test
  • ran make test-all command

guilload avatar Dec 27 '21 18:12 guilload

As discussed offline with @fulmicoton, we currently don't have a pressing need for this API. We will revisit when necessary.

guilload avatar Dec 28 '21 09:12 guilload

Reopened. See #1631.

guilload avatar Jun 14 '22 21:06 guilload

This PR is ready for review.

I have introduced two new metastore APIs, which completely clears the checkpoints (keys are removed)

  • reset_index_checkpoint(index_id)
  • reset_source_checkpoint(index_id, source_id)

I have introduced two new CLI commands:

  • index reset-checkpoint --index <index ID>
  • source reset-checkpoint --index <index ID> --source <source ID>

guilload avatar Jul 08 '22 14:07 guilload

PTAL!

guilload avatar Jul 21 '22 18:07 guilload

FYI, in relation to this as part of the Kafka offset storage I've been working on metastore level changes to move checkpoints into a separate table on Postgres backed metastores. As part of that I'm adding two APIs, one of which would overlap with this.

    /// Retrieves the current checkpoints for an index
    ///
    /// Returns a `IndexCheckpoint` for the provided `index_id` and optionally filtered by
    /// `source_id`, `resource` and `group_id`.
    /// If no checkpoint has been stored yet the result will be an empty `IndexCheckpoint`.
    async fn index_checkpoint(
        &self,
        index_id: &str,
        source_id: Option<String>,
        resource: Option<String>,
        group_id: Option<String>,
    ) -> MetastoreResult<IndexCheckpoint>;

    /// Removes the current checkpoints for an index
    ///
    /// Removes all checkpoints that match the provided `index_id` and optionally filtered by
    /// `source_id`, `resource` and `group_id`.
    /// If no checkpoint has been stored it will not be considered an error.
    async fn delete_index_checkpoint(
        &self,
        index_id: &str,
        source_id: Option<String>,
        resource: Option<String>,
        group_id: Option<String>,
    ) -> MetastoreResult<()>;

kstaken avatar Jul 21 '22 20:07 kstaken