unexpected exception from store.columns()

[subbyte]

While doing a small refactoring for commands.py in Kestrel, I find an existing unit test does not behave as I thought.

The unit test: https://github.com/opencybersecurityalliance/kestrel-lang/blob/develop/tests/test_timestamped.py#L85

The huntflow to reproduce the exception and the stack:

conns = GET network-traffic
        FROM https://raw.githubusercontent.com/opencybersecurityalliance/kestrel-lang/develop/tests/test_bundle.json
	    WHERE dst_ref.value NOT ISSUBSET '192.168.0.0/16'

grp_conns = GROUP conns BY dst_ref.value WITH COUNT(dst_ref) AS count

ts_grp_conns = TIMESTAMPED(grp_conns)

The error when running the huntflow:

Traceback (most recent call last):
  File "/home/subx/venv/kestrel-dev/lib/python3.11/site-packages/firepit/sqlitestorage.py", line 161, in _do_execute
    cursor.execute(query)
sqlite3.OperationalError: no such column: grp_conns.id

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/subx/venv/kestrel-dev/lib/python3.11/site-packages/kestrel/codegen/summary.py", line 97, in get_variable_entity_count
    columns = variable.store.columns(variable.entity_table)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/subx/venv/kestrel-dev/lib/python3.11/site-packages/firepit/sqlitestorage.py", line 290, in columns
    cursor = self._execute(stmt)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/subx/venv/kestrel-dev/lib/python3.11/site-packages/firepit/sqlitestorage.py", line 185, in _execute
    return self._do_execute(statement, cursor=cursor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/subx/venv/kestrel-dev/lib/python3.11/site-packages/firepit/sqlitestorage.py", line 168, in _do_execute
    raise InvalidAttr(m) from e
firepit.exceptions.InvalidAttr: invalid attribute: grp_conns.id

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/subx/venv/kestrel-dev/bin/kestrel", line 9, in <module>
    runpy.run_module("kestrel", run_name="__main__")
  File "<frozen runpy>", line 229, in run_module
  File "<frozen runpy>", line 88, in _run_code
  File "/home/subx/venv/kestrel-dev/lib/python3.11/site-packages/kestrel/__main__.py", line 32, in <module>
    outputs = session.execute(huntflow)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/subx/venv/kestrel-dev/lib/python3.11/site-packages/kestrel/session.py", line 274, in execute
    return self._execute_ast(ast)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/subx/venv/kestrel-dev/lib/python3.11/site-packages/kestrel/session.py", line 427, in _execute_ast
    output_var_struct, display = execute_cmd(stmt, self)
                                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/subx/venv/kestrel-dev/lib/python3.11/site-packages/kestrel/codegen/commands.py", line 102, in wrapper
    return func(stmt, session)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/subx/venv/kestrel-dev/lib/python3.11/site-packages/kestrel/codegen/commands.py", line 66, in wrapper
    var_struct = new_var(
                 ^^^^^^^^
  File "/home/subx/venv/kestrel-dev/lib/python3.11/site-packages/kestrel/symboltable/variable.py", line 128, in new_var
    return VarStruct(
           ^^^^^^^^^^
  File "/home/subx/venv/kestrel-dev/lib/python3.11/site-packages/kestrel/symboltable/variable.py", line 38, in __init__
    self.length = get_variable_entity_count(self)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/subx/venv/kestrel-dev/lib/python3.11/site-packages/kestrel/codegen/summary.py", line 102, in get_variable_entity_count
    raise MissingEntityAttribute(table_name, attr) from e
kestrel.exceptions.MissingEntityAttribute: [ERROR] MissingEntityAttribute: variable "grp_conns" does not have required attribute "id"
remove transform or specify different variable in the Kestrel command.

The strange: it is the function store.columns() that hit the InvalidAttr exception in firepit when running ts_grp_conns = TIMESTAMPED(grp_conns). Should it just return all columns?

[pcoccoli]

I agree the stack trace is confusing, but grp_conns is an aggregate and therefore cannot be used with TIMESTAMPED. That transform needs to join the private __contains table and the observed-data table using id, but aggregates do not have an id.