embucket-labs icon indicating copy to clipboard operation
embucket-labs copied to clipboard

DISCUSSION query history: Sql Text filter

Open YaroslavLitvinov opened this issue 8 months ago • 0 comments

This is a sub-issue of main issue #428 , for adding Sql Text Filter into /ui/queries handler.

  • GET /ui/queries request have sqlText parameter
  • /ui/queries response contains results with matched sqlText sub-strings

Implementation options, decide which one to select:

  • No optimization - iterate every stored query and match sub-string on sql_text field. This is a full scan approach, on big loads it should heavily hit performance.
  • Do a custom optimization on top of slatedb. This allows making search by keys, instead of fetching and comparing values. When adding queries it creates a bunch of reference keys for every word in sql_text as below, to be used as dictionary when searching words. In worksheets we already used reference keys, and it is relatively easy to implement (issue #426 ).
  //   Add index for two words: "create", "hello". No need updating values when adding new queries.
  /idx/create/qh/key-timestamp
  /idx/create/qh/key2-timestamp
  /idx/hello/qh/key-timestamp
  that allows to iterate on range of results [/idx/create : /idx/create/qh/max-id ]
  • Create a process of importing data into tables for every N-minutes like it's done in reference implementation, and then query with any filter options supported by our query engine. And then access data using views:
    • information_schema.QUERY_HISTORY
    • information_schema.QUERY_HISTORY_BY_USER
    • ACCOUNT_USAGE.QUERY_HISTORY @camuel may be interested.

YaroslavLitvinov avatar Apr 22 '25 19:04 YaroslavLitvinov