embucket-labs
embucket-labs copied to clipboard
DISCUSSION query history: Sql Text filter
This is a sub-issue of main issue #428 , for adding Sql Text Filter into /ui/queries handler.
- GET
/ui/queriesrequest havesqlTextparameter /ui/queriesresponse contains results with matchedsqlTextsub-strings
Implementation options, decide which one to select:
- No optimization - iterate every stored query and match sub-string on sql_text field. This is a full scan approach, on big loads it should heavily hit performance.
- Do a custom optimization on top of
slatedb. This allows making search by keys, instead of fetching and comparing values. When adding queries it creates a bunch of reference keys for every word in sql_text as below, to be used as dictionary when searching words. In worksheets we already used reference keys, and it is relatively easy to implement (issue #426 ).
// Add index for two words: "create", "hello". No need updating values when adding new queries.
/idx/create/qh/key-timestamp
/idx/create/qh/key2-timestamp
/idx/hello/qh/key-timestamp
that allows to iterate on range of results [/idx/create : /idx/create/qh/max-id ]
- Create a process of importing data into tables for every N-minutes like it's done in reference implementation, and then query with any filter options supported by our query engine. And then access data using views:
- information_schema.QUERY_HISTORY
- information_schema.QUERY_HISTORY_BY_USER
- ACCOUNT_USAGE.QUERY_HISTORY @camuel may be interested.