Task: Make a list of Zebra dependencies

[mpguerra]

Motivation

When we engage a team to audit Zebra, we want to provide them with a number of lists outlining our dependencies and which of those we would like to be in scope for the audit.

Task

We will need 3 lists:

• [ ] Direct ECC dependencies that are in scope for the audit (most will be out of scope as they have already been audited)
• [ ] Direct Zebra dependencies that are in scope as they are consensus or security critical
• [ ] All other direct dependencies which are out of scope, including the ECC ones.

We can create the full list of all production Zebra dependencies with a command like:

cargo tree --depth 1 --prefix none --locked --edges normal --workspace --exclude zebra-client --exclude zebra-test --features sentry,journald,filter-reload,prometheus

The options to cargo tree are listed here: https://doc.rust-lang.org/cargo/commands/cargo-tree.html

ECC dependencies are usually from https://github.com/zcash Consensus-critical dependencies are used to implement specific consensus rules, for example, zcash_script.

[teor2345]

I've noticed that we're not automatically updating some dependencies using dependabot.

In some cases, this appears to be a bug (or incorrect documentation) for dependabot. Closing automated dependabot PRs seems to stop all dependency updates, not just the selected minor or major version.

If any of these dependencies are in scope for the audit, we might want to upgrade them before tagging the release candidate.

I reopened dependabot PRs for these dependencies:

• color-eyre v0.6.1 -> v0.6.2
• metrics-exporter-prometheus v0.9.0 -> v0.11.0
• metrics v0.18.1 -> v0.20.1
• sha2 v0.9.9 -> v0.10.5 (just to check)
• tracing-subscriber v0.3.11 -> v0.3.15
• tracing v0.1.31 -> v0.1.36

These aren't all release blockers - we need to check the list created by this ticket.

These dependencies are deliberately kept in sync with the versions used by other dependencies:

• aes v0.7.5 -> v0.8.1
• gumdrop v0.7 -> v0.8.1 (abscissa)
• secp256k1 v0.21.3 -> v0.24.0
• sha2 v0.9.9 -> v0.10.5

These dependencies are too much work to upgrade everywhere:

• pin-project v0.4.29 -> v1.0.12
• proptest v0.10.1 -> v1.0.0

The list was created using cargo upgrade --workspace 2>&1 1>/dev/null | sort -u on 6 September 2022. Then I removed some dependencies we decided not to upgrade.

It's tricky to re-enable dependabot upgrades on closed dependency PRs. The only way to do that is by re-opening the ignored PR.

[dconnolly]

Closing automated dependabot PRs seems to stop all dependency updates, not just the selected minor or major version.

Well that's annoying.

[teor2345]

Closing automated dependabot PRs seems to stop all dependency updates, not just the selected minor or major version.

Well that's annoying.

It seems like they recently updated the dependabot message to match its actual behaviour:

[upbqdn]

Closing automated dependabot PRs seems to stop all dependency updates, not just the selected minor or major version.

I manually used the "Close with comment" button on PRs #5221 and #5224, and this is what I got:

It looks like we'll get new updates based on what Dependabot said in the image, no?

I was going to reopen the two PRs to see what happens, but it looks like I can't do it.

[mpguerra]

Hey team! Please add your planning poker estimate with Zenhub @arya2 @conradoplg @dconnolly @gustavovalverde @oxarbitrage @teor2345 @upbqdn

[oxarbitrage]

So, for the list of zebra production dependencies, i think some tables can be a good way to present them.

Dependencies could change from now to the date we send the list to the auditors so i made a small an ugly script who convert output from the cargo tree command into an md table: https://github.com/oxarbitrage/cargo-tree-table/blob/main/src/main.rs

The output is below, we might want to do change to the script instead of a hardcoded list.

---

Zebra dependencies

tower-batch v0.2.30

Name	Version
futures	0.3.24
futures-core	0.3.24
pin-project	1.0.12
rayon	1.5.3
tokio	1.21.2
tokio-util	0.7.4
tower	0.4.13
tracing	0.1.36
tracing-futures	0.2.5

tower-fallback v0.2.30

Name	Version
futures-core	0.3.24
pin-project	0.4.30
tower	0.4.13
tracing	0.1.36

zebra-chain v1.0.0-beta.15

Name	Version
aes	0.7.5
bech32	0.9.1
bitflags	1.3.2
bitvec	1.0.1
blake2b_simd	1.0.0
blake2s_simd	1.0.0
bls12_381	0.7.0
bs58	0.4.0
byteorder	1.4.3
chrono	0.4.22
displaydoc	0.2.3
ed25519-zebra	3.0.0
equihash	0.2.0
fpe	0.5.1
futures	0.3.24
group	0.12.0
halo2_proofs	0.2.0
hex	0.4.3
humantime	2.1.0
incrementalmerkletree	0.3.0
itertools	0.10.5
jubjub	0.9.0
lazy_static	1.4.0
orchard	0.2.0
primitive-types	0.11.1
proptest	0.10.1
proptest-derive	0.3.0
rand	0.8.5
rand_chacha	0.3.1
rand_core	0.6.4
rayon	1.5.3
redjubjub	0.5.0
ripemd	0.1.3
secp256k1	0.21.3
serde	1.0.145
serde-big-array	0.4.1
serde_with	2.0.1
sha2	0.9.9
static_assertions	1.1.0
subtle	2.4.1
thiserror	1.0.37
tokio	1.21.2
tracing	0.1.36
uint	0.9.4
x25519-dalek	1.2.0
zcash_encoding	0.1.0
zcash_history	0.3.0
zcash_note_encryption	0.1.0
zcash_primitives	0.7.0
zebra-test	1.0.0-beta.15

zebra-consensus v1.0.0-beta.15

Name	Version
bellman	0.13.1
blake2b_simd	1.0.0
bls12_381	0.7.0
chrono	0.4.22
dirs	4.0.0
displaydoc	0.2.3
futures	0.3.24
futures-util	0.3.24
halo2_proofs	0.2.0
jubjub	0.9.0
lazy_static	1.4.0
metrics	0.20.1
once_cell	1.15.0
orchard	0.2.0
proptest	0.10.1
proptest-derive	0.3.0
rand	0.8.5
rayon	1.5.3
serde	1.0.145
thiserror	1.0.37
tokio	1.21.2
tower	0.4.13
tower-batch	0.2.30
tower-fallback	0.2.30
tracing	0.1.36
tracing-futures	0.2.5
zcash_proofs	0.7.1
zebra-chain	1.0.0-beta.15
zebra-script	1.0.0-beta.15
zebra-state	1.0.0-beta.15

zebra-network v1.0.0-beta.15

Name	Version
bitflags	1.3.2
byteorder	1.4.3
bytes	1.2.1
chrono	0.4.22
futures	0.3.24
hex	0.4.3
humantime-serde	1.1.1
indexmap	1.9.1
lazy_static	1.4.0
metrics	0.20.1
ordered-map	0.4.2
pin-project	1.0.12
proptest	0.10.1
proptest-derive	0.3.0
rand	0.8.5
rayon	1.5.3
regex	1.6.0
serde	1.0.145
thiserror	1.0.37
tokio	1.21.2
tokio-stream	0.1.10
tokio-util	0.7.4
tower	0.4.13
tracing	0.1.36
tracing-error	0.2.0
tracing-futures	0.2.5
zebra-chain	1.0.0-beta.15

zebra-node-services v1.0.0-beta.15

Name	Version
zebra-chain	1.0.0-beta.15

zebra-rpc v1.0.0-beta.15

Name	Version
chrono	0.4.22
futures	0.3.24
hex	0.4.3
hyper	0.14.20
indexmap	1.9.1
jsonrpc-core	18.0.0
jsonrpc-derive	18.0.0
jsonrpc-http-server	18.0.0
num_cpus	1.13.1
serde	1.0.145
serde_json	1.0.85
tokio	1.21.2
tower	0.4.13
tracing	0.1.36
tracing-futures	0.2.5
zebra-chain	1.0.0-beta.15
zebra-network	1.0.0-beta.15
zebra-node-services	1.0.0-beta.15
zebra-state	1.0.0-beta.15

zebra-script v1.0.0-beta.15

Name	Version

zebra-state v1.0.0-beta.15

Name	Version

zebra-utils v1.0.0-beta.15

Name	Version
color-eyre	0.6.2
hex	0.4.3
serde_json	1.0.85
structopt	0.3.26
tinyvec	1.6.0
tracing-error	0.2.0
tracing-subscriber	0.3.15
zebra-chain	1.0.0-beta.15
zebra-consensus	1.0.0-beta.15
zebra-state	1.0.0-beta.15

zebrad v1.0.0-beta.15

Name	Version
abscissa_core	0.5.2
atty	0.2.14
chrono	0.4.22
color-eyre	0.6.2
dirs	4.0.0
futures	0.3.24
gumdrop	0.7.0
humantime	2.1.0
humantime-serde	1.1.1
hyper	0.14.20
indexmap	1.9.1
lazy_static	1.4.0
log	0.4.17
metrics	0.20.1
metrics-exporter-prometheus	0.11.0
num-integer	0.1.45
pin-project	1.0.12
rand	0.8.5
rayon	1.5.3
sentry	0.27.0
sentry-tracing	0.27.0
serde	1.0.145
thiserror	1.0.37
tinyvec	1.6.0
tokio	1.21.2
toml	0.5.9
tower	0.4.13
tracing	0.1.36
tracing-appender	0.2.2
tracing-error	0.2.0
tracing-futures	0.2.5
tracing-journald	0.3.0
tracing-subscriber	0.3.15
zebra-chain	1.0.0-beta.15
zebra-consensus	1.0.0-beta.15
zebra-network	1.0.0-beta.15
zebra-node-services	1.0.0-beta.15
zebra-rpc	1.0.0-beta.15
zebra-state	1.0.0-beta.15

[teor2345]

So, for the list of zebra production dependencies, i think some tables can be a good way to present them.

Dependencies could change from now to the date we send the list to the auditors so i made a small an ugly script who convert output from the cargo tree command into an md table: https://github.com/oxarbitrage/cargo-tree-table/blob/main/src/main.rs

@oxarbitrage thanks for doing this!

Here's what we can do next:

Put it in the zebra repository

We might want to put the script somewhere so any developer can run it. Can you open a PR to add it to https://github.com/ZcashFoundation/zebra/tree/main/zebra-utils ?

We'll also want to keep a record of the final automatically generated list, and the final manually categorised list. We could put it in an audit directory in https://github.com/ZcashFoundation/zebra/tree/main/book/src/dev ?

Fix the automatic list

I can see some issues with the automatically generated list:

• the zebra-script and zebra-state dependencies are empty
• the proptest dependencies are test-only dependencies, we might need to adjust the command to skip them
• we need one list of unique dependencies, it doesn't matter which zebra crates use each dependency

Manually split the list

After we have the automatic list, we need to manually split it into:

• Direct ECC dependencies that are in scope for the audit (most will be out of scope as they have already been audited)
• Direct Zebra dependencies that are in scope as they are consensus or security critical
• All other direct dependencies which are out of scope, including the ECC ones.

ECC dependencies are usually from https://github.com/zcash Consensus-critical dependencies are used to implement specific consensus rules, for example, zcash_script.

Let me know if you need help with any of this!

[teor2345]

We'll also need to manually adjust the version of the zcash_proofs dependency, because it is a git dependency: https://github.com/ZcashFoundation/zebra/blob/9849d14f41a5c014bab90eed851a6eb0162883e2/zebra-consensus/Cargo.toml#L39

[oxarbitrage]

the zebra-script and zebra-state dependencies are empty

I think this might be because all the dependencies are duplicated.

[teor2345]

the zebra-script and zebra-state dependencies are empty

I think this might be because all the dependencies are duplicated.

We're missing rocksdb and zcash_script, which are unique dependencies of zebra-state and zebra-script.

[oxarbitrage]

We're missing rocksdb and zcash_script, which are unique dependencies of zebra-state and zebra-script.

I think it might be the cargo tree command. They were empty also when running it in the command line. Will try more with the command variations.

[oxarbitrage]

I updated the program so it can have a link to the version we are using of each dependency (where possible): https://github.com/oxarbitrage/cargo-tree-table/blob/main/outputs/zebra.md

The manual split i am still confused with in scope and out of scope.

I am also wondering, why doing this list when we are supposed to be paying a huge amount of for an audit and we don't have idea if they will request this from us ?

[mpguerra]

The manual split i am still confused with in scope and out of scope.

A very simple example of dependencies that would be out of scope would be crates used for logging and/or tracing as they are not consensus critical, e.g the tracing crate.

I am also wondering, why doing this list when we are supposed to be paying a huge amount of for an audit and we don't have idea if they will request this from us ?

We want to tell the auditors which dependencies they should ignore and which they might want to look at more closely. We don't want them concentrating on dependencies that are trivial or which are not used in critical ways.

[oxarbitrage]

I have another list, i think is still to big so we might want to remove entire sections and/or dependencies individually:

https://gist.github.com/oxarbitrage/033bcf655212dd3b57136d01e70ab472#file-zebra-deps-md

From the automatic list i manually removed all the duplicates, categorized different and removed dependencies.

Removed dependencies: https://gist.github.com/oxarbitrage/033bcf655212dd3b57136d01e70ab472#file-zebra-deps-removed-md

[teor2345]

I'm focused on the release candidate right now, so I might not be able to review these lists in detail.

We'll need to split the ECC dependencies into the dependencies they have got audited, and dependencies they haven't.

We'll also need to do the same thing for shared ZF dependencies like ed25519-zebra, redjubjub, and reddsa (or whatever we're using to implement redpallas).

[oxarbitrage]

I was talking about this with @teor2345 yesterday and we decided to make a pull request/beanch for this so different people can make changes. https://github.com/ZcashFoundation/zebra/pull/5395