sql icon indicating copy to clipboard operation
sql copied to clipboard

[FEATURE] Stabilize preferred SQL implicit sorting method

Open Swiddis opened this issue 4 months ago • 2 comments

Is your feature request related to a problem? Coming from #3061, there seems to be a few places in-code where we implicitly sort records when ORDER BY clauses aren't supplied by a user. Currently we use _doc, a per-shard incrementing counter for each indexed document. This works fine for single-shard indices, but causes unstable sorting in some cases when there are multiple shards involved. As a workaround, I'm adding a secondary sort by _id in cases where it's known that this causes issues, but this has a large performance impact (~4x latency in local testing).

What solution would you like? There should be a single source of truth for what the "implicit sorting" key is, and that implicit sort should be guaranteed unique per-document. This can probably go in a helper library somewhere, I'm just not quite acquainted enough with the plugin to know where that would be.

What alternatives have you considered?

  • Sort just by ID, throw out _doc: this might be faster than trying to sort by two keys, but throws out all ordering.
  • Sort by @timestamp: Not guaranteed to be present on all indices anymore.
  • Some other field that bundles the rough chronology of _doc with information on shards: doesn't yet exist to my knowledge.

Do you have any additional context? See the comments on #3061

Swiddis avatar Oct 09 '24 18:10 Swiddis