Document recommended procedure for reinitializing package loads on Zeek upgrade

[dspruell-s01]

macOS Catalina 10.15.6 Python 3.8.5 zeek 3.2.1 zkg 2.3.1

Facing some challenges in zkg with every Zeek update, not certain if I'm missing something obvious or if opportunity to improve process.

Is there a recommended and straightforward procedure for reinitializing a package installation after a Zeek update? After every Zeek version update I seem to run into situation that the package configuration is present in the previous installation but needs to be set up again in the new installation. At the very least, it seems like this requires a zkg autoconfig to update settings for zkg:

$ zkg autoconfig
Set "script_dir" config option to: /usr/local/Cellar/zeek/3.2.1/share/zeek/site ?
(previous value: /usr/local/Cellar/zeek/3.2.0/share/zeek/site) [Y/n] 
Set "plugin_dir" config option to: /usr/local/Cellar/zeek/3.2.1/lib/zeek/plugins ?
(previous value: /usr/local/Cellar/zeek/3.2.0/lib/zeek/plugins) [Y/n] 
Set "zeek_dist" config option to: /tmp/zeek-20200910-79596-149i5z5 ?
(previous value: /tmp/zeek-20200817-20501-x7lmyy) [Y/n] 
Successfully wrote config file to /Users/dspruell/.zkg/config

From here it's not clear to me what to do to restore the configuration of installed packages so that they're loaded in the updated install (site) config. Packages are installed (in the sense that zkg has installed them and is aware of them):

$ zkg list
zeek/hosom/file-extraction (installed: 2.0.3) - Extract files from network traffic with Zeek.
zeek/micrictor/smbfp (installed: master) - A package to create a fingerprint of SMB clients
zeek/salesforce/hassh (installed: master) - HASSH is used to identify specific Client and Server SSH implementations.
zeek/salesforce/ja3 (installed: master) - JA3 creates 32 character SSL client fingerprints and logs them as a field in ssl.log.
zeek/sethhall/zeek-log-all-http-headers (installed: v1.0.0) - Add all HTTP headers and values to the HTTP log.
zeek/theparanoids/rdfp (installed: master) - The script will create a new log which will log the details which build the fingerprint and some additional information.

They are not loaded in the new configuration however, as doing so requires the site/local.zeek script to be updated (@load packages). Then site/packages/packages.zeek needs to be configured to @load the scripts (this file is managed by zkg). This is a part that is unclear to me. Presumably since packages are installed, they simply need to be loaded to the config. One might anticipate a one-shot command to do that, which zkg doesn't seem to provide, so:

$ zkg load $(zkg list |awk '{print $1}')
The package "zeek/hosom/file-extraction" does not contain scripts to load.
The package "zeek/micrictor/smbfp" does not contain scripts to load.
The package "zeek/salesforce/hassh" does not contain scripts to load.
The package "zeek/salesforce/ja3" does not contain scripts to load.
The package "zeek/sethhall/zeek-log-all-http-headers" does not contain scripts to load.
The package "zeek/theparanoids/rdfp" does not contain scripts to load.

"does not contain scripts to load" is confusing here.

So acknowledging I'm probably lost along the way, two questions:

1. Is zkg missing a convenience option that could modify local.zeek to load packages, and then enable (re)loading of installed scripts in packages.zeek? zkg manages packages.zeek, so presumably it could also activate a single line in local.zeek. Optionally, could it store the desired configuration of which scripts to enable and/or related settings to configure in local.zeek for a quick one-shot command that could restore the desired config post-upgrade.
2. Is there a repeatable process that simplifies this case of restoring the zeek package installation and loaded script configuration after a minor zeek upgrade?

[jsiwek]

If Zeek upgrades are going into new install roots like that, I'd try something like this:

zkg bundle zeek-packages.bundle
# Upgrade Zeek at this point...
zkg autoconfig
zkg unbundle zeek-packages.bundle
echo "@load packages" >> ${new_zeek_install_root}/share/zeek/site/local.zeek

[ckreibich]

For now we should probably cover this in the zkg documentation. At the moment we don't really say how/whether zkg treats multiple parallel Zeek installations. (Shout if I missed something!)

Two scenarios that might intuitively come to the user's mind are

• zkg just figures it out and internally tracks things separately for different installations
• zkg only supports one, so if you switch versions, best to start over

Seems we currently exist sort of in-between, because you can run zkg autoconfig on top of an existing installation, and it (nicely) warns you of the changes you make to the existing config. But zkg's internal state about existing installations remains (yell if I'm off here), and so for example zkg list continues to list packages.

If we start to bundle zkg with the Zeek distribution, this might be win-win: if we keep all of its state co-located with the installed Zeek tree, multiple Zeek installations would also naturally obtain independent zkg package state.

Fwiw, I've also found this kind of thing confusing in other package managers, including pip.