quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

Push API - Add mechanism to check status

Open PSeitz opened this issue 2 years ago • 1 comments

Issue Outline

The Push API currently returns the number of docs received, but this does not mean the documents are indexed, they are merely stored in the rocks-db.

It could be that the documents had a wrong format or other errors. Currently there is no way for the user to know the status of committed documents, except checking the server logs or check if they appear in search.

Async API design

The Push API is essential a async API. A typical pattern for async APIs is to return an id handle to the user with status code 202 when the documents have been securely stored. This handle can then be used to check the actual status of the submitted documents.

The endpoint to check could be a plain http returning some status in json, or websocket-like server sent events.

Async API pattern: https://docs.microsoft.com/en-us/azure/architecture/patterns/async-request-reply

PSeitz avatar May 13 '22 11:05 PSeitz

The async API is nice but may be hard to use in practice without a native Quickwit client or support from log collectors. I'm in favor of a sync API with throttling:

  1. ensure index exists
  2. parse docs
  3. ensure docs match index schema
  4. store tantivy docs in WAL
  5. ack or (partially or fully) reject the batch

Things get tricky when updating index schemas. I have a few ideas in mind but this is probably worthy of a design doc.

guilload avatar Jun 27 '22 09:06 guilload

See #3743 insead.

guilload avatar Aug 14 '23 15:08 guilload