use ARM crypto extensions to improve ZFS encryption performance

[albertofustinoni]

System information

Type	Version/Name
Distribution Name	Armbian Buster
Distribution Version	21.05.2
Linux Kernel	5.10.34-meson64
Architecture	aarch64 (S905X3/Cortex-A55)
ZFS Version	2.0.3-1~bpo10+1
SPL Version	2.0.3-1~bpo10+1

Describe the problem you're observing

I am running ZFS on an Odroid HC4 and I'm noticing significantly worse performance with ZFS native encryption VS creating unencrypted pool over a dm-crypto block device created via LUKS.

We are talking between 40 and 70MB/s and 100% CPU usage for native crypto depending on which cypher I choose vs saturating gigabit ethernet at ~70% CPU usage when copying via Samba.

The Cortex A55 supports ARM's crypto extensions and I'm assuming dm-crypto takes advantage of them given these results

cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       477493 iterations per second for 256-bit key
PBKDF2-sha256     878204 iterations per second for 256-bit key
PBKDF2-sha512     447344 iterations per second for 256-bit key
PBKDF2-ripemd160  305529 iterations per second for 256-bit key
PBKDF2-whirlpool  114573 iterations per second for 256-bit key
argon2i       4 iterations, 477274 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id      4 iterations, 479560 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
#     Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b       535.0 MiB/s       591.1 MiB/s
    serpent-cbc        128b        44.9 MiB/s        49.5 MiB/s
    twofish-cbc        128b        70.0 MiB/s        74.9 MiB/s
        aes-cbc        256b       464.2 MiB/s       547.2 MiB/s
    serpent-cbc        256b        44.9 MiB/s        49.5 MiB/s
    twofish-cbc        256b        70.0 MiB/s        74.8 MiB/s
        aes-xts        256b       566.8 MiB/s       566.2 MiB/s
    serpent-xts        256b        46.3 MiB/s        49.9 MiB/s
    twofish-xts        256b        73.6 MiB/s        76.0 MiB/s
        aes-xts        512b       526.9 MiB/s       526.4 MiB/s
    serpent-xts        512b        46.3 MiB/s        49.9 MiB/s
    twofish-xts        512b        73.6 MiB/s        76.0 MiB/s

Is this a case of ZFS not being able to do the same? Or is this related to the kernel breaking crypto acceleration for ZFS a while ago? Was the fix only for x86/AES-NI?