[Feature Request] Fetch signed block header (`LedgerInfoWithSignatures`) for any arbitrary block height in a PFN

[wwared]

Prior refs

This issue is motivated by the development of a self-contained light client for Aptos (context), and some previous issues discussing this are #11824, #10747 and #7401.

Motivation

This issue refers specifically to the need of a light client to fetch a signed Block Header for the block height it wants to generate a Merkle proof of inclusion of some value. It's necessary to not only generate a SparseMerkleProof for an arbitrary version, but also fetch the corresponding LedgerInfoWithSignatures for that specific version so the light client can verify the provenance of the state root Merkle hash used in the SparseMerkleProof.

Our exploration

Looking through the AptosDB schema files, we've identified the following places that store LedgerInfoWithSignatures directly for easy retrieval, however none of them seem sufficient for this:

• The ledger_info schema uses Epoch numbers to key specific LedgerInfoWithSchema values, keeping one per epoch (the most recent, I believe, up to the last block in the epoch), so this is not able to give us the LedgerInfoWithSignatures for a specific version or height
• The block_by_version and block_info schemas lets us find the corresponding block height for a version as well as the BlockInfo struct for any arbitrary block height, but the BlockInfo struct does not contain any signature data, so it is not sufficient by itself
• The methods in the DbReader trait (implemented in aptosdb_reader) don't have any methods that lets us get the LedgerInfoWithSignatures for a specific block height or version, since these methods focus on either getting the latest/most recent LedgerInfoWithSignatures (get_latest_ledger_info_option), or getting the StateProof containing both a LedgerInfoWithSignatures and an EpochChangeProof for a specific version (get_epoch_ending_ledger_info and get_state_proof_with_ledger_info). It doesn't seem like any methods let us query for arbitrary LedgerInfoWithSignatures for any arbitrary block
• There exists the Quorum Store DB, and its quorum_certificate schema allows us to map a block hash to a QuorumCert struct, which does contain the LedgerInfoWithSignatures! However: the Quorum Store DB does not seem to be available on every regular Public Full Node, only Validators, so querying this DB does not seem to be the correct choice (here is the initialization code that only starts the corresponding services for Validators). I've also attempted dumping both the Consensus DB and the Quorum Store DB via the admin interface debug endpoints, but for my non-validator full node those returned an error saying the DB was unavailable
  • There seems to be a new in-development "consensus observer" feature that seems to enable some (or both) of the Consensus DB and Quorum Store DB for PFNs that enable it in the config, but it's unclear to me if this feature is meant to support this usecase or something else

Next steps

I wasn't able to find any direct and intended way of retrieving signed block headers for arbitrary blocks witnessed by a full node. Is there something I'm missing in regards to how this data is stored in the node's DB, or some other way of fetching historical LedgerInfoWithSignatures data for any arbitrary block height? If not, could support for this be added?

I understand that for most cases, the node only needs to keep track of LedgerInfoWithSignatures that are necessary for processing Epoch changes, as well as its most recent value, but I presume that since the full node ends up witnessing all of the LedgerInfoWithSignatures during the process of keeping up with the network, that this data would be stored somewhere and could be retrieved in some way.