[crypto+move] add support for Blake2b-256 in Move

[blasrodri]

Taken a lot of inspiration from https://github.com/aptos-labs/aptos-core/pull/4181#issue-1372293035

Description

Add BLAKE2B-256 hashing in Move, to support folks writing bridge contracts to other platforms, as per conversations with @JoshLind and @alinush.

• it is feature-gated
• it has (reasonable) gas costs for the new hash functions, marked with optional
• it bumps up the gas feature version

Test Plan

This PR merely exports an existing hash function crate (i.e.: blake2_rfc), which are is believed to be trustworthy implementation as it's used in Substrate (https://crates.io/crates/sp-core-hashing/4.0.0/dependencies). Therefore, we only made sure the native implementation give the expected hash values for b"testing" and b"testing again" since I couldn't get the output of an empty string in the online hash generator.

---

This change is 

[alinush]

Thank you for the PR @blasrodri!

There are two flavors of Blake: Blake2b, Blake2s (see https://www.blake2.net/).

Can you please update the PR title and all of the code (e.g., function names, gas parameter names, comments, etc.) to clearly indicate which one you are implementing?

[alinush]

Thank you for the update @blasrodri! Can you include in the PR's description the performance impact of going from blake2 crate to blake2-rfc?

[blasrodri]

Thank you for the update @blasrodri! Can you include in the PR's description the performance impact of going from blake2 crate to blake2-rfc?

Oh it's just because i checked the Substrate implemention and it uses blake2_rfc. But ill do a quick benchmark, and post the results. So that we can pick the better one ☺️

[blasrodri]

blake2 seems to be faster than blake2-rfc

use blake2::{
    digest::{Update, VariableOutput},
    Blake2bVar,
};
use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn blake2_blake2b_256(data: &[u8]) -> Vec<u8> {
    // Blake2bVarCore::new_with_params(salt, persona, key_size, output_size)
    let mut hasher = Blake2bVar::new(32).unwrap();
    hasher.update(data);
    let mut buf = vec![0u8; 32];
    hasher.finalize_variable(&mut buf).unwrap();
    buf
}

fn blake2_rfc_blake2b_256(data: &[u8]) -> Vec<u8> {
    blake2_blake2b_256(data).to_vec()
}

fn criterion_benchmark(c: &mut Criterion) {
    assert_eq!(
        blake2_blake2b_256(b"testing"),
        blake2_rfc_blake2b_256(b"testing")
    );
    c.bench_function("blake2_blake2b_256", |b| {
        b.iter(|| blake2_blake2b_256(black_box(b"testing")))
    });
    c.bench_function("blake2_rfc_blake2b_256", |b| {
        b.iter(|| blake2_rfc_blake2b_256(black_box(b"testing")))
    });
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);

blas@f1:~/foss/blak2-benchmarks$ cargo bench
   Compiling blak2-benchmarks v0.1.0 (/home/blas/foss/blak2-benchmarks)
    Finished bench [optimized] target(s) in 0.79s
     Running unittests src/main.rs (target/release/deps/blak2_benchmarks-c07cbf1492157f71)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/blake2.rs (target/release/deps/blake2-ee4b1cddd724fc46)
blake2_blake2b_256      time:   [123.94 ns 124.07 ns 124.21 ns]
                        change: [-0.2375% +0.0861% +0.3560%] (p = 0.59 > 0.05)
                        No change in performance detected.
Found 10 outliers among 100 measurements (10.00%)
  8 (8.00%) high mild
  2 (2.00%) high severe

blake2_rfc_blake2b_256  time:   [142.25 ns 142.37 ns 142.51 ns]
                        change: [-0.7492% -0.6241% -0.4932%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild

blas@f1:~/foss/blak2-benchmarks$ cargo bench
    Finished bench [optimized] target(s) in 0.02s
     Running unittests src/main.rs (target/release/deps/blak2_benchmarks-c07cbf1492157f71)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/blake2.rs (target/release/deps/blake2-ee4b1cddd724fc46)
blake2_blake2b_256      time:   [122.38 ns 122.51 ns 122.64 ns]
                        change: [-1.5252% -1.3730% -1.2095%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

blake2_rfc_blake2b_256  time:   [139.35 ns 139.51 ns 139.71 ns]
                        change: [-1.8661% -1.6701% -1.4750%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

[alinush]

blake2 seems to be faster than blake2-rfc

use blake2::{
    digest::{Update, VariableOutput},
    Blake2bVar,
};
use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn blake2_blake2b_256(data: &[u8]) -> Vec<u8> {
    // Blake2bVarCore::new_with_params(salt, persona, key_size, output_size)
    let mut hasher = Blake2bVar::new(32).unwrap();
    hasher.update(data);
    let mut buf = vec![0u8; 32];
    hasher.finalize_variable(&mut buf).unwrap();
    buf
}

fn blake2_rfc_blake2b_256(data: &[u8]) -> Vec<u8> {
    blake2_blake2b_256(data).to_vec()
}

fn criterion_benchmark(c: &mut Criterion) {
    assert_eq!(
        blake2_blake2b_256(b"testing"),
        blake2_rfc_blake2b_256(b"testing")
    );
    c.bench_function("blake2_blake2b_256", |b| {
        b.iter(|| blake2_blake2b_256(black_box(b"testing")))
    });
    c.bench_function("blake2_rfc_blake2b_256", |b| {
        b.iter(|| blake2_rfc_blake2b_256(black_box(b"testing")))
    });
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);

blas@f1:~/foss/blak2-benchmarks$ cargo bench
   Compiling blak2-benchmarks v0.1.0 (/home/blas/foss/blak2-benchmarks)
    Finished bench [optimized] target(s) in 0.79s
     Running unittests src/main.rs (target/release/deps/blak2_benchmarks-c07cbf1492157f71)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/blake2.rs (target/release/deps/blake2-ee4b1cddd724fc46)
blake2_blake2b_256      time:   [123.94 ns 124.07 ns 124.21 ns]
                        change: [-0.2375% +0.0861% +0.3560%] (p = 0.59 > 0.05)
                        No change in performance detected.
Found 10 outliers among 100 measurements (10.00%)
  8 (8.00%) high mild
  2 (2.00%) high severe

blake2_rfc_blake2b_256  time:   [142.25 ns 142.37 ns 142.51 ns]
                        change: [-0.7492% -0.6241% -0.4932%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild

blas@f1:~/foss/blak2-benchmarks$ cargo bench
    Finished bench [optimized] target(s) in 0.02s
     Running unittests src/main.rs (target/release/deps/blak2_benchmarks-c07cbf1492157f71)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/blake2.rs (target/release/deps/blake2-ee4b1cddd724fc46)
blake2_blake2b_256      time:   [122.38 ns 122.51 ns 122.64 ns]
                        change: [-1.5252% -1.3730% -1.2095%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

blake2_rfc_blake2b_256  time:   [139.35 ns 139.51 ns 139.71 ns]
                        change: [-1.8661% -1.6701% -1.4750%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Thank you @blasrodri! Let's stick to blake2 in that case.

Could you please add your benchmarks to this hash.rs file in crates/aptos-crypto/benches/?

This will help me estimate gas costs more accurately.

[blasrodri]

blake2 seems to be faster than blake2-rfc

use blake2::{
    digest::{Update, VariableOutput},
    Blake2bVar,
};
use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn blake2_blake2b_256(data: &[u8]) -> Vec<u8> {
    // Blake2bVarCore::new_with_params(salt, persona, key_size, output_size)
    let mut hasher = Blake2bVar::new(32).unwrap();
    hasher.update(data);
    let mut buf = vec![0u8; 32];
    hasher.finalize_variable(&mut buf).unwrap();
    buf
}

fn blake2_rfc_blake2b_256(data: &[u8]) -> Vec<u8> {
    blake2_blake2b_256(data).to_vec()
}

fn criterion_benchmark(c: &mut Criterion) {
    assert_eq!(
        blake2_blake2b_256(b"testing"),
        blake2_rfc_blake2b_256(b"testing")
    );
    c.bench_function("blake2_blake2b_256", |b| {
        b.iter(|| blake2_blake2b_256(black_box(b"testing")))
    });
    c.bench_function("blake2_rfc_blake2b_256", |b| {
        b.iter(|| blake2_rfc_blake2b_256(black_box(b"testing")))
    });
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);

blas@f1:~/foss/blak2-benchmarks$ cargo bench
   Compiling blak2-benchmarks v0.1.0 (/home/blas/foss/blak2-benchmarks)
    Finished bench [optimized] target(s) in 0.79s
     Running unittests src/main.rs (target/release/deps/blak2_benchmarks-c07cbf1492157f71)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/blake2.rs (target/release/deps/blake2-ee4b1cddd724fc46)
blake2_blake2b_256      time:   [123.94 ns 124.07 ns 124.21 ns]
                        change: [-0.2375% +0.0861% +0.3560%] (p = 0.59 > 0.05)
                        No change in performance detected.
Found 10 outliers among 100 measurements (10.00%)
  8 (8.00%) high mild
  2 (2.00%) high severe

blake2_rfc_blake2b_256  time:   [142.25 ns 142.37 ns 142.51 ns]
                        change: [-0.7492% -0.6241% -0.4932%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild

blas@f1:~/foss/blak2-benchmarks$ cargo bench
    Finished bench [optimized] target(s) in 0.02s
     Running unittests src/main.rs (target/release/deps/blak2_benchmarks-c07cbf1492157f71)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/blake2.rs (target/release/deps/blake2-ee4b1cddd724fc46)
blake2_blake2b_256      time:   [122.38 ns 122.51 ns 122.64 ns]
                        change: [-1.5252% -1.3730% -1.2095%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

blake2_rfc_blake2b_256  time:   [139.35 ns 139.51 ns 139.71 ns]
                        change: [-1.8661% -1.6701% -1.4750%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Thank you @blasrodri! Let's stick to blake2 in that case.

Could you please add your benchmarks to this hash.rs file in crates/aptos-crypto/benches/?

This will help me estimate gas costs more accurately.

I made a mistake on the previous script. Now, I've added the two implementations on the hash.rs bench, and these are the results:

hash/BLAKE2B-256-blake2-crate/0                                                                             
                        time:   [156.02 ns 156.21 ns 156.39 ns]
                        thrpt:  [0.0000   B/s 0.0000   B/s 0.0000   B/s]
                 change:
                        time:   [+0.3640% +0.5418% +0.7898%] (p = 0.00 < 0.05)
                        thrpt:  [-0.7836% -0.5389% -0.3627%]
                        Change within noise threshold.
Found 32 outliers among 1000 measurements (3.20%)
  16 (1.60%) high mild
  16 (1.60%) high severe
hash/BLAKE2B-256-blake2-rfc-crate/0                                                                             
                        time:   [125.81 ns 125.89 ns 125.97 ns]
                        thrpt:  [0.0000   B/s 0.0000   B/s 0.0000   B/s]
                 change:
                        time:   [+0.2890% +0.4097% +0.5219%] (p = 0.00 < 0.05)
                        thrpt:  [-0.5192% -0.4080% -0.2882%]
                        Change within noise threshold.

hash/BLAKE2B-256-blake2-crate/1                                                                             
                        time:   [171.15 ns 171.28 ns 171.40 ns]
                        thrpt:  [5.5640 MiB/s 5.5681 MiB/s 5.5720 MiB/s]
                 change:
                        time:   [+10.641% +10.729% +10.818%] (p = 0.00 < 0.05)
                        thrpt:  [-9.7620% -9.6898% -9.6175%]
                        Performance has regressed.
Found 26 outliers among 1000 measurements (2.60%)
  22 (2.20%) high mild
  4 (0.40%) high severe
hash/BLAKE2B-256-blake2-rfc-crate/1                                                                             
                        time:   [134.56 ns 134.72 ns 134.90 ns]
                        thrpt:  [7.0693 MiB/s 7.0788 MiB/s 7.0874 MiB/s]
                 change:
                        time:   [-1.2537% -1.0534% -0.8347%] (p = 0.00 < 0.05)
                        thrpt:  [+0.8417% +1.0646% +1.2696%]
                        Change within noise threshold.
Found 11 outliers among 1000 measurements (1.10%)
  8 (0.80%) high mild
  3 (0.30%) high severe

[alinush]

Sweet! I forgot to ask the most obvious question: are these numbers for x86_64?

[blasrodri]

Sweet! I forgot to ask the most obvious question: are these numbers for x86_64?

Yes ^_^

blas@f1:~$ uname -m
x86_64

[alinush]

One more request, since you are adding new gas parameters, you will need to bump up the LATEST_GAS_FEATURE_VERSION in aptos-move/aptos-gas/src/gas_meter.rs (e.g., see this).

Right, @vgao1996? (Sorry, I keep forgetting this.)

[alinush]

Sweet! I forgot to ask the most obvious question: are these numbers for x86_64?

Yes ^_^

blas@f1:~$ uname -m
x86_64

Another request: could you re-run benchmarks both for SHA2-256 and your newly-added Blake2b function and post the results here? This will allow us to calibrate the gas costs a little better.

The SHA2-256 ones are also in aptos-crypto/benches/hash.rs

[blasrodri]

Sweet! I forgot to ask the most obvious question: are these numbers for x86_64?

Yes ^_^

blas@f1:~$ uname -m
x86_64

Another request: could you re-run benchmarks both for SHA2-256 and your newly-added Blake2b function and post the results here? This will allow us to calibrate the gas costs a little better.

The SHA2-256 ones are also in aptos-crypto/benches/hash.rs

Sure! Will do it today

[blasrodri]

SHA2

hash/SHA2-256/0         time:   [71.395 ns 71.552 ns 71.720 ns]                             
                        thrpt:  [0.0000   B/s 0.0000   B/s 0.0000   B/s]
                 change:
                        time:   [+4.6839% +4.9618% +5.2726%] (p = 0.00 < 0.05)
                        thrpt:  [-5.0086% -4.7273% -4.4743%]
                        Performance has regressed.
Found 53 outliers among 1000 measurements (5.30%)
  19 (1.90%) high mild
  34 (3.40%) high severe
hash/SHA2-256/8         time:   [72.274 ns 72.323 ns 72.374 ns]                             
                        thrpt:  [105.42 MiB/s 105.49 MiB/s 105.56 MiB/s]
                 change:
                        time:   [+0.9191% +1.1247% +1.3492%] (p = 0.00 < 0.05)
                        thrpt:  [-1.3312% -1.1122% -0.9108%]
                        Change within noise threshold.
Found 4 outliers among 1000 measurements (0.40%)
  2 (0.20%) high mild
  2 (0.20%) high severe
hash/SHA2-256/16        time:   [67.068 ns 67.177 ns 67.301 ns]                              
                        thrpt:  [226.72 MiB/s 227.14 MiB/s 227.51 MiB/s]
                 change:
                        time:   [-5.5021% -5.3377% -5.1669%] (p = 0.00 < 0.05)
                        thrpt:  [+5.4484% +5.6387% +5.8225%]
                        Performance has improved.
Found 94 outliers among 1000 measurements (9.40%)

BLAKE2B

hash/BLAKE2B-256-blake2-crate/2                                                                             
                        time:   [169.94 ns 170.12 ns 170.33 ns]
                        thrpt:  [11.198 MiB/s 11.212 MiB/s 11.223 MiB/s]
                 change:
                        time:   [-2.9107% -2.7912% -2.6447%] (p = 0.00 < 0.05)
                        thrpt:  [+2.7166% +2.8713% +2.9980%]
                        Performance has improved.
Found 23 outliers among 1000 measurements (2.30%)
  10 (1.00%) low mild
  7 (0.70%) high mild
  6 (0.60%) high severe

hash/BLAKE2B-256-blake2-rfc-crate/8                                                                             
                        time:   [132.83 ns 132.88 ns 132.94 ns]
                        thrpt:  [57.392 MiB/s 57.415 MiB/s 57.439 MiB/s]
                 change:
                        time:   [-4.1867% -3.7555% -3.3995%] (p = 0.00 < 0.05)
                        thrpt:  [+3.5191% +3.9020% +4.3696%]
                        Performance has improved.
Found 7 outliers among 1000 measurements (0.70%)

[davidiw]

One more request, since you are adding new gas parameters, you will need to bump up the LATEST_GAS_FEATURE_VERSION in aptos-move/aptos-gas/src/gas_meter.rs (e.g., see this).

Right, @vgao1996? (Sorry, I keep forgetting this.)

I'm not sure why this would be required. There's nothing that can leverage this within the code. I don't think we update this without there being something that would depend on previous behavior. So your previous change on it shouldn't have been done afaiu.

[github-actions[bot]]

Forge is running suite land_blocking on 2e67931cdfc79fce446f04bc409a6363e71a944d

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite land_blocking success on 2e67931cdfc79fce446f04bc409a6363e71a944d

performance benchmark with full nodes : 6303 TPS, 6247 ms latency, 12400 ms p99 latency,(!) expired 2260 out of 2693920 txns
Test Ok

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[davidiw]

This branch has conflicts that must be resolved

Please rebase and fix on your end. I can potentially help, but I think there might be some gnarly changes.

[alinush]

I'm coordinating with Blas on re-basing and merging. No worries @davidiw.

[github-actions[bot]]

Forge is running suite land_blocking on 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite land_blocking success on 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c

performance benchmark with full nodes : 6665 TPS, 5953 ms latency, 9000 ms p99 latency,no expired txns
Test Ok

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c

Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7284 TPS, 5331 ms latency, 7000 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c
compatibility::simple-validator-upgrade::single-validator-upgrade : 4545 TPS, 8820 ms latency, 12600 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c
compatibility::simple-validator-upgrade::half-validator-upgrade : 4296 TPS, 9750 ms latency, 12800 ms p99 latency,no expired txns
4. upgrading second batch to new version: 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c
compatibility::simple-validator-upgrade::rest-validator-upgrade : 6189 TPS, 6201 ms latency, 9200 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 3fbc1403be0e5377c9012c5ec44b7a8b42f97e6c passed
Test Ok

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking