launcher
launcher copied to clipboard
[Kolide ATC] Construct KATC tables and add support for Firefox extension data
Builds on https://github.com/kolide/launcher/pull/1761
This PR adds config parsing for KATC tables and constructing those tables.
As a proof-of-concept, it implements one table type and a couple data processing steps, with the end result of supporting querying Firefox extension data (sqlite file + data is compressed via snappy + data is serialized with StructuredClone).
I think will be easiest to review the changes in this order:
pkg/osquery/table/table.go-- this is the entrypoint to theee/katcpackage, and touches the changes made in the previous PRee/katc/config.go-- this is the parsing and construction of the KATC tablesee/katc/table.go-- this is the generation of results when a query is run against a KATC tableee/katc/sqlite.go-- this is querying a specific backend (in this case, a sqlite database) on behalf of a KATC table (this function is called byee/katc/table.go)ee/katc/snappy.go-- this is a fairly simple row transform step (called byee/katc/table.goon the sqlite results)ee/katc/deserialize_firefox.go-- this is a significantly more complicated row transform step (called byee/katc/table.goon the sqlite results that have already been snappy decompressed); ultimately it's deserializing a JS object into a Golang map
An example KATC config:
{
"my_example_table": {
"source_type": "sqlite",
"source": "/some/%/path/to/db.sqlite",
"platform": "darwin",
"columns": ["someKeyInsideCompressedSerializedJSONObject"],
"query": "SELECT data FROM object_data JOIN object_store ON (object_data.object_store_id = object_store.id) WHERE object_store.name=\"some-test-table\";",
"row_transform_steps": ["snappy", "structured_clone"]
}
}
Work TBD in separate PRs:
- Backfill tests: https://github.com/kolide/launcher/pull/1766
- Transform camel case columns to snake case: https://github.com/kolide/launcher/pull/1765
- Add support for indexeddb/leveldb tables: begun in https://github.com/kolide/launcher/pull/1767
Perhaps this is a silly suggestion, but do we want to consider the capability to copy/read/delete for DB's that are locked while open? Perhaps an optional boolean column eg. read_from_copy that puts the DB in /tmp or elsewhere?
Perhaps this is a silly suggestion, but do we want to consider the capability to copy/read/delete for DB's that are locked while open? Perhaps an optional boolean column eg.
read_from_copythat puts the DB in/tmpor elsewhere?
We should explore that use case. There may be other sqlite options that let us open locked files. immutable specifically
@FritzX6 not a silly suggestion! I had thought about keeping that in the implementation details instead of the config (e.g. for leveldb+indexeddb we always want to copy the db, we shouldn't allow configuration otherwise) -- maybe there's a stronger way to require that. I will look into the immutable suggestion from seph also.