Epic: Support for RPC calls used by mining pools

[mpguerra]

Motivation

We would like mining pools to be able to use zebrad to:

1. generate a block template for mining, and
2. broadcast a newly-mined block to the Zcash network.

Our use cae assumption is that the miner will use Zebra to generate the block template, mine the block outside of Zebra, then use Zebra to broadcast the block to the Zcash network.

Miners should be able to use Zebra as a drop-in replacement for zcashd (i.e. the interface with Zebra should be compatible with zcashd’s).

Prerequisites

We did an initial scope and design in Zebra Mining RPC scope.pdf.

We also want to do this analysis and design work:

• Get a list of all RPCs called by the s-nomp mining pool software to mine blocks (excluding payments), and where they are called from
• Work out which parameters and fields we need for each RPC
• Create a data flow diagram that shows how Zebra handles each RPC, so we know if we need state, mempool, or verifier changes
• Decide which tests we need and which tests are out of scope

Scope

To begin with this will require us to support the following RPC calls:

• [ ] #5235
• [ ] #5236
• [ ] #5268 (#4967)
• [ ] #5302
• [ ] #5303

Implement ZIP 317:

• [ ] #5335
• [ ] #5336
• In getblocktemplate: select transactions based on ZIP-317

Fix the following bugs:

• [ ] #2631
• [x] The block part of #483
• [ ] #5376
• [ ] #5386

And test the RPCs:

• [ ] #5304

We also want to disable these RPCs in production zebrad builds, until they have been tested:

• [ ] #5305

Optional Tasks

These changes increase miner profits, but they are not required:

Allow transactions that spend UTXOs created in the same block, to increase fees:

• Send mempool UTXOs in verifier requests
• Sort transactions that create UTXOs before transactions that spend them in getblocktemplate

Speed up verification of new blocks, to decrease orphaned blocks and lost subsidies:

• Automatically verify transactions that are already in the mempool (could be really quick to implement), or
• Cache expensive proof & signature verifier results (like #1968)

Out of Scope

• Mining functionality
• Mine blocks on testnet during the tests
  • Deploy (non-mining) testnet nodes each time we merge to main or tag a release
• Node wallet RPCs used by mining pools or miners to manage funds (e.g. receiving block rewards, making mining pool payouts)

[teor2345]

@mpguerra you can see the full list of required RPCs and Zebra bug fixes here: https://hackmd.io/gjxd0c4oTp-ZB81PtHdU2w

This is a draft list, it won't be final until we've found some open-source mining pool coordination software to test against.

[mpguerra]

So is the planning done for this already or do we need to do some more?

I'm keen to try to do this asynchronously using github discussions. I've started one for this purpose here: https://github.com/ZcashFoundation/zebra/discussions/5267

If it is not needed because all of the planning is done we can just close it...

[teor2345]

So is the planning done for this already or do we need to do some more?

Planning is incomplete, I've added some missing tasks to this ticket and the discussion.

[teor2345]

@mpguerra I added a list of optional tasks that will increase miner profits (but I didn't create tickets yet).

[teor2345]

I checked s-nomp to see which RPCs it actually uses:

Current scope:

• getblocktemplate - s-nomp and node-stratum-pool
• submitblock - node-stratum-pool
• getblockcount - node-stratum-pool
• getblocksubsidy - might be optional in node-stratum-pool, falls back to getblocktemplate
• getblockhash - no

Other mining RPCs:

• getinfo - uses fields that Zebra hasn't implemented yet
  • https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L441
• getblock(verbosity=1) - uses fields that Zebra hasn't implemented yet
  • https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L710
• validateaddress - s-nomp and node-stratum-pool
  • https://github.com/s-nomp/s-nomp/blob/8567a4c649ab6f5ea37ac3c3901f2d9c2597199a/libs/poolWorker.js#L140
  • https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L388
• getdifficulty and getmininginfo - node-stratum-pool
  • https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L388

Might be optional:

• getpeerinfo - only used for sync progress
  • https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L173
• getnetworkinfo - only used for payment processing, or when getinfo is not available:
  • https://github.com/s-nomp/s-nomp/blob/1fe93a6fabfd35a542e3c0fa606c3728bedaad9e/libs/paymentProcessor.js#L380
  • https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L397

Not required:

• prioritisetransaction - not used by s-nomp or node-stratum-pool
• getlocalsolps, getnetworkhashps, getnetworksolps - not used by s-nomp or node-stratum-pool

We should also check what RPCs other mining pools use.

[mpguerra]

For an MVP, I would stick to the RPCs that are actually in the list of mining RPCs in the rpc docs, so:

Other mining RPCs:

* [getinfo](https://zcash.github.io/rpc/getinfo.html) - uses fields that Zebra hasn't implemented yet
  
  * https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L441

This does not seem like it's strictly mining related

* [getblock(verbosity=1)](https://zcash.github.io/rpc/getblock.html) - uses fields that Zebra hasn't implemented yet
  
  * https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L710

This does not seem like it's strictly mining related

* [validateaddress](https://zcash.github.io/rpc/validateaddress.html) - `s-nomp` and `node-stratum-pool`
  
  * https://github.com/s-nomp/s-nomp/blob/8567a4c649ab6f5ea37ac3c3901f2d9c2597199a/libs/poolWorker.js#L140
  * https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L388

This does not seem like it's strictly mining related

* [getdifficulty](https://zcash.github.io/rpc/getdifficulty.html) and [getmininginfo](https://zcash.github.io/rpc/getmininginfo.html) - `node-stratum-pool`
  
  * https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L388

getdifficulty does not seem like it's strictly mining related, getmininginfo does seem to be mining related, we should probably add this one to the list

Might be optional:

* [getpeerinfo](https://zcash.github.io/rpc/getpeerinfo.html) - only used for sync progress
  
  * https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L173

* [getnetworkinfo](https://zcash.github.io/rpc/getnetworkinfo.html) - only used for payment processing, or when getinfo is not available:
  
  * https://github.com/s-nomp/s-nomp/blob/1fe93a6fabfd35a542e3c0fa606c3728bedaad9e/libs/paymentProcessor.js#L380
  * https://github.com/s-nomp/node-stratum-pool/blob/d86ae73f8ff968d9355bb61aac05e0ebef36ccb5/lib/pool.js#L397

Again, none of these seem to be strictly mining related

Not required:

* prioritisetransaction - no

* getlocalsolps, getnetworkhashps, getnetworksolps - no

We should also check what RPCs other mining pools use.

These are in the list in the docs so how do we know that they are not required? Are they not used by s-nomp? If so I'm happy to exclude from the MVP.

[teor2345]

For an MVP, I would stick to the RPCs that are actually in the list of mining RPCs in the rpc docs

How can we be sure that the RPC docs are accurate for Zcash mining?

As far as I can see, they contain a lot of legacy RPCs that aren't used for Zcash, or aren't required when using getblocktemplate. Some of the documentation and fields of individual RPCs are also out of date.

If we don't implement the RPCs it expects, the mining pool software will fail to run. Can we set the scope based on the mining pool software code or discussions with engineers, rather than outdated documentation?

These are in the list in the docs so how do we know that they are not required? Are they not used by s-nomp? If so I'm happy to exclude from the MVP.

prioritisetransaction, getlocalsolps, getnetworkhashps, and getnetworksolps are not used by s-nomp or its mining pool dependency node-stratum-pool. This is another reason I think the documentation is outdated.

[mpguerra]

All done here 👏 🎉