gRPC message size limit of 4MB is too restrictive

[conorsch]

Describe the bug We are limited to sending gRPC messages with a max size of 4MB, which is insufficient for at least two use cases:

• handling large genesis files (containing updated discord allocations)
• working on summonerd

To Reproduce

1. Update discord allocations
2. Run new-testnet.sh script
3. Run smoke tests.

Expected behavior Application proceeds past genesis handling and smoke tests past.

Additional context Example error message:

Sep 18 16:49:18.666 INFO penumbra_view::worker: view worker error e=status: OutOfRange, message: "Error, message length too large: found 5347639 bytes, the limit is: 4194304 bytes", details: [], metadata: MetadataMap { headers: {} }

The tonic docs state that max_decoding_message_size can be overridden on a per-client basis, but it's not obvious to me how we'd customize that config attribute in our generated proto files.

[redshiftzero]

See commit a088e87a4548ecea1ba48b17fca12a059b0c308d which used the max_decoding_message_size function to address the same issue

[conorsch]

We solved this for the summonerd functionality. Let's see if we can resolve the issue for large genesis files, as well.

[conorsch]

Started on this in https://github.com/penumbra-zone/penumbra/pull/3191. Supporting large genesis files was not much of a problem. The changes, however, broke the smoke tests, due to an obscure h2 bug about "stream error: error writing a body to connection: send stream capacity unexpectedly closed". I believe this is caused by 1) the balances request using a streaming api; and 2) the pcli implementation using a direct call to the balances method without instantiating an intermediate client, on which we can hang the max message size attributes. I'm going to try the max message size conversion one more time, from a clean slate, targeting the absolute minimal diff, to isolate where the breakage is happening.

[conorsch]

Talked through the challenge with @zbuc and got some helpful pointers. I've adapted the view code to use client constructions throughout, which allows us to add the message size customizations. Still wrestling with how to refactor the view.as_mut here: https://github.com/penumbra-zone/penumbra/blob/c767608254597971e018e75e3ac802cec7d6acce/crates/bin/pcli/src/main.rs#L45-L54 Once that's resolved, CI should be green again.

[conorsch]

This issue is no longer pressing, so downgrading to Future. We currently have a limitation that genesis files cannot be larger than 4MB. That's OK for now.

[hdevalence]

Is the limitation that the genesis file can't be larger than 4MB, or that the resulting CompactBlock can't be larger than 4MB?