zed
zed copied to clipboard
lake index: Get type indexes working again
@mccanne could use some guidance on design for this.
Also, should type indexes allow multiple-typed indexes?
Related to this, the following text was present in a retired "lake design" document (see #3803). It has been established that this was really a pending to-do, so this issue tracks its ultimate implementation and corresponding updates to docs.
A type rule indicates that all values of any field of a specified type
be indexed where the type signature uses Zed type syntax.
For example,
zed index create IndexGroupEx type ip
creates a rule that indexes all IP addresses appearing in fields of type `ip`
in the index group `IndexGroupEx`.
Note that as of current Zed commit 28f95a3, for a user it seems like it's possible to create such indexes. E.g.:
$ zed -version
Version: v1.2.0-64-g28f95a3c
$ zed index create IndexGroupEx type ip
IndexGroupEx
rule 2FN8RkkgYCwsvFYIYpVvxlg569M type ip
$ zed create zeek
pool created: zeek 2FN8TMvn5asIvdTeLnGbyQ0X7mi
$ zed -use zeek load *
(26/26) 44.71MB/44.71MB 0B/s 100.00%
2FN8USotSZtHIbU6UBmePwcnsBz committed
$ zed index update
2FN8V0I79edGCaOGrY3SyY9RY9F committed
$ find ~/lake
/Users/phil/lake
...
/Users/phil/lake/2FN8TMvn5asIvdTeLnGbyQ0X7mi/index
/Users/phil/lake/2FN8TMvn5asIvdTeLnGbyQ0X7mi/index/2FN8RkkgYCwsvFYIYpVvxlg569M
/Users/phil/lake/2FN8TMvn5asIvdTeLnGbyQ0X7mi/index/2FN8RkkgYCwsvFYIYpVvxlg569M/2FN8UC5iStrQ0uLIv6Twpv9HsSh.zng
...
$ zq -z /Users/phil/lake/2FN8TMvn5asIvdTeLnGbyQ0X7mi/index/2FN8RkkgYCwsvFYIYpVvxlg569M/2FN8UC5iStrQ0uLIv6Twpv9HsSh.zng | head -10
{key:2.22.230.64,count:9(uint64)}
{key:4.53.160.75,count:12(uint64)}
{key:5.9.78.71,count:3(uint64)}
However, I also confirmed with @mattnibs that such indexes are not currently leveraged during queries.
Closing because #4972 removed lake indexes.