Create a bundle auditing tool (endo unbundle)

[kriskowal]

For the purposes of verifying the content and integrity of a bundle, as motivated by the need to audit governance proposals with Agoric chains, we need a tool that can verify the integrity of a bundle and extract it to the file system so an auditor can inspect the full working set of sources, without the distraction of adjacent files from the original source tree, notably in the absence of source maps.

Currently, we have @endo/check-package which provides the implementation of the integrity check. We also have most of the tools necessary for extraction between @endo/zip and JSON.parse. We just need a CLI veneer. That could be an endo subcommand.

This would be a substitute for the manual advice for validating a bundle:

• Verify that jq -r .moduleFormat bundle.json is endoZipBase64
• Extract the zip file jq -r .endoZipBase64 bundle.json | base64 -d > bundle.zip (or the governance proposal can just refer to a zip file, or even a tarball)
• unzip bundle.zip
• sha512sum compartmetn-map.json must match the proposed bundle identifier!
• In compartment-map.json, every module description must include a sha512 that matches the sha512sum of the corresponding .js.pre.json module file.
• You can extract the readable source from each of the precompiled modules with jq -r '.__syncModuleProgram__' compartment/module.pre.js.json

[dckc]

I prototyped a web-based tool for exploring bundles after they are put on chain:

• https://github.com/Agoric/agoric-sdk/discussions/8416

[dckc]

stretch goal: bundle diff... it would be nice if I went from marshal 0.6.1 to 0.6.2, it told me what the differences are, rather than saying "only in b2: 0.6.2"

[dckc]

More motivation:

• 2013-12: struggling to reproduce assets: https://github.com/Agoric/agoric-3-proposals/pull/41#issuecomment-1852492922

[dckc]

different sizes for same module?!

I was asked how do we see the code of the core-evals that ran in #69 Enable stTIA vaults.

Some of the code is there on the blockchain, but some of it is included by reference to a bundle:

const manifestBundleRef = {
  bundleID:
    "b1-903e41a7c448a41b456298404a1c32c69302574209c6a5228723ed19e2dd99f2a693641196445bc27a90e19e1dfadfe6b3d9c9a93f080ffa33a70908e5af4fff",
};

That bundle is available in /bundles of https://github.com/0xpatrickdev/agoric-vault-collateral-proposal/commit/f9f60a65ac400319e28a0897ffd81256e663300f and attached here for convenience:

• expected-903e.zip

I tried to reproduce the bundle; the entry module should be https://github.com/Agoric/agoric-sdk/blob/4f9a171b79cd2a3ca5576ea452a4c8c595d5deb8/packages/inter-protocol/src/proposals/addAssetToVault.js , according to fairly careful notes in https://github.com/dckc/agoric-vault-collateral-proposal/releases/tag/v0.13.0-alpha3 (more detail: https://github.com/0xpatrickdev/agoric-vault-collateral-proposal/issues/9#issuecomment-1949621008 )

But when I bundle that code, what I actually get is:

• actual-1e13.zip
  • b1-1e1385adb83612c9e4192875e605f82b7e6af003d0a89d4327df370bef25060e4959d916353da9c1a098d17e78bbcbbe5bd9dbf482e23e0a5dc6e1327c8f47c7.

I used earlier work on a bundle explorer and made summaries of them to compare.

What I found is the same module name (package version, filename) with different sizes:

Name	Size (expected)	Size (actual)
@agoric/assert-v0.6.0/src/assert.js	3795	3807
@agoric/base-zone-v0.1.0/heap.js	351	364
@agoric/base-zone-v0.1.0/src/heap.js	2740	2751
...
@agoric/inter-protocol-v0.16.1/src/proposals/addAssetToVault.js	14726	14736

[dckc]

oh. these are not released packages. so the version numbers can stay still while the files change. oh boy.

[dckc]

um... never mind: wrong agoric-sdk version due to ambient agoric CLI from $PATH.

see also https://github.com/0xpatrickdev/agoric-vault-collateral-proposal/issues/9#issuecomment-1949644873

[dckc]

stretch goal: bundle diff...

I just discovered:

• diffoscope: in-depth comparison of files, archives, and directories

diffoscope tries to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human-readable form to compare them. It can compare two tarballs, ISO images, or PDF just as easily.

[kriskowal]

Here’s an “endo unbundle” implementation that will work on Mac.

function undo() {
  mkdir -p "$1" && jq -r .endoZipBase64 | base64 -d | tar xC "$1"
}

Usage:

undo bundle < bundle.json