launcher icon indicating copy to clipboard operation
launcher copied to clipboard

[Kolide ATC] Construct KATC tables and add support for Firefox extension data

Open RebeccaMahany opened this issue 1 year ago • 4 comments

Builds on https://github.com/kolide/launcher/pull/1761

This PR adds config parsing for KATC tables and constructing those tables.

As a proof-of-concept, it implements one table type and a couple data processing steps, with the end result of supporting querying Firefox extension data (sqlite file + data is compressed via snappy + data is serialized with StructuredClone).

I think will be easiest to review the changes in this order:

  1. pkg/osquery/table/table.go -- this is the entrypoint to the ee/katc package, and touches the changes made in the previous PR
  2. ee/katc/config.go -- this is the parsing and construction of the KATC tables
  3. ee/katc/table.go -- this is the generation of results when a query is run against a KATC table
  4. ee/katc/sqlite.go -- this is querying a specific backend (in this case, a sqlite database) on behalf of a KATC table (this function is called by ee/katc/table.go)
  5. ee/katc/snappy.go -- this is a fairly simple row transform step (called by ee/katc/table.go on the sqlite results)
  6. ee/katc/deserialize_firefox.go -- this is a significantly more complicated row transform step (called by ee/katc/table.go on the sqlite results that have already been snappy decompressed); ultimately it's deserializing a JS object into a Golang map

An example KATC config:

{
    "my_example_table": {
        "source_type": "sqlite",
        "source": "/some/%/path/to/db.sqlite",
        "platform": "darwin",
        "columns": ["someKeyInsideCompressedSerializedJSONObject"],
        "query": "SELECT data FROM object_data JOIN object_store ON (object_data.object_store_id = object_store.id) WHERE object_store.name=\"some-test-table\";",
        "row_transform_steps": ["snappy", "structured_clone"]
    }
}

Work TBD in separate PRs:

  • Backfill tests: https://github.com/kolide/launcher/pull/1766
  • Transform camel case columns to snake case: https://github.com/kolide/launcher/pull/1765
  • Add support for indexeddb/leveldb tables: begun in https://github.com/kolide/launcher/pull/1767

RebeccaMahany avatar Jun 26 '24 20:06 RebeccaMahany

Perhaps this is a silly suggestion, but do we want to consider the capability to copy/read/delete for DB's that are locked while open? Perhaps an optional boolean column eg. read_from_copy that puts the DB in /tmp or elsewhere?

FritzX6 avatar Jun 27 '24 13:06 FritzX6

Perhaps this is a silly suggestion, but do we want to consider the capability to copy/read/delete for DB's that are locked while open? Perhaps an optional boolean column eg. read_from_copy that puts the DB in /tmp or elsewhere?

We should explore that use case. There may be other sqlite options that let us open locked files. immutable specifically

directionless avatar Jun 27 '24 13:06 directionless

@FritzX6 not a silly suggestion! I had thought about keeping that in the implementation details instead of the config (e.g. for leveldb+indexeddb we always want to copy the db, we shouldn't allow configuration otherwise) -- maybe there's a stronger way to require that. I will look into the immutable suggestion from seph also.

RebeccaMahany avatar Jun 27 '24 14:06 RebeccaMahany

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Jul 02 '24 20:07 CLAassistant