Enable Signer Setup Easily via AWS CDK

[AshtonStephens]

Summary

Enable signer setup easily via AWS CDK. This ticket isn't related directly to the code in this repository, but is part of the user facing artifacts required for a swift ratification of SIP-021.

Loosely, we want to have a stacks-node deployment template specified in this Amazon managed repository here: aws-blockchain-node-runners that has the capacity to run with a signer, stacks-node, and bitcoin node.

This ticket includes all the work necessary to collaborate with AWS.

Tasks

1. Work with Amazon / AWS to define what they want in their review processes
2. Define project ownership between AWS and ecosystem resources after the initial merge
3. Write a document on the resources and design of the infrastructure we want deployed to AWS
4. Create changes in aws-blockchain-node-runners to deploy resources above & test and verify that the infrastructure is deployed correctly
5. Work with AWS to merge those changes to aws-blockchain-node-runners

Timeline

1. [x] Initial scoping and discussions with AWS team (1 day)
2. [x] Design and plan Stacks AWS node-runner (~~1.2~ -> ~2 weeks)
   1. [x] Manually set up Stacks node types in AWS and document steps (~1~ -> 3 days)
   2. [x] Write draft design document and review with engineers (1 day)
   3. [x] Automate Stacks Follower node; requires the most boilerplate, debugging, and testing (~3~ -> 4 days)
   4. [x] Write final design document for AWS cloud solution (2 days)
   5. [ ] Review and alter design document internally and with AWS (2 days)
3. [ ] Implement AWS node-runners (~1 week)
   1. [x] Finalize Stacks Follower node based on feedback from final design doc (1 day)
   2. [ ] Create stacks Signer node; Signer node has never been set up, probably won't be fast. Will require clarity work as well (4 days)
4. [ ] Review process (hopeful ~1.5 weeks)
   1. [ ] AWS team will need to review the changes for any security concerns, they say this could be a week to a month depending on how we architect it, so this estimate is based on our architecture choices being idiomatic
   2. [ ] Stacks ecosystem developers will need to review and verify the code

Artifiacts

• [ ] A document planning and designing the template resources
• [ ] A merged PR to aws-blockchain-node-runners with changes from the design document.

[AshtonStephens]

Created https://github.com/aws-samples/aws-blockchain-node-runners/issues/42 in the AWS related GH repository.

[AshtonStephens]

My best guess is that it will take around four weeks to go from nothing to the Stacks node-runner being merged into aws-blockchain-node-runners, but that's aggressive in our expectations of the AWS team's reviews. I'll keep this ticket updated and will incorporate timeline changes.

This estimate means we should have an easy to spin up Stacks follower, signer, and miner some time around March 1st. The public Testnet program will start March 8th, so in the worst case scenario we need a working version of the miner and signer nodes by March 8th in a CDK template regardless of whether those templates are merged into the node-runner repository's main branch.

Estimated Breakdown and Timeline

Now out of date. up to date timeline is in the description above

1. [x] Initial scoping and discussions with AWS team (1 day)
2. [ ] Design and plan Stacks AWS node-runner (~1.2 weeks)
   1. [x] Manually set up Stacks node types in AWS and document steps (1 day)
   2. [x] Write design document for AWS cloud solution (3 days)
   3. [ ] Review and alter design document internally and with AWS (2 days)
3. [ ] Implement AWS node-runners (~1 week)
   1. [ ] Create Stacks Follower node; requires most boilerplate, debugging, and testing (3 days)
   2. [ ] Create Stacks Miner node; not much to change from the previous node (1 day)
   3. [ ] Create stacks Signer node; Signer node has never been set up, probably won't be fast (2 days)
4. [ ] Review process (hopeful ~1.5 weeks)
   1. [ ] AWS team will need to review the changes for any security concerns, they say this could be a week to a month depending on how we architect it, so this estimate is based on our architecture choices being idiomatic
   2. [ ] Stacks ecosystem developers will need to review and verify the code

Edit

I've adjusted the order and timing above to include automating the follower before writing the full design document and allowing more time for creating the initial automation. I'm discovering some quirks of the CDK constructs and the EC2 instance as I go through the automation in addition to the manual setup.

[AshtonStephens]

Still working on the manual setup, but I'm writing the document as I go. The miner and follower configuration are going to be very similar with minor tweaks to the metrics they publish and their configuration files. The configurations will need to largely exist within AWS Secrets Manager because the miner has a private keys within it, and the Stacker / Signer node will as well.

The next step in my manual setup is to connect to the Bitcoin node. It makes the most sense to have the Stacks nodes connect to the AWS managed Bitcoin nodes to use instead of hosting their own Bitcoin nodes. Unfortunately, all rpc calls to the Bitcoin nodes require sigv4 authentication. We need to utilize the sigv4 proxy to sign the rpc calls en-route to the amazon managed blockchain.

• https://aws.amazon.com/managed-blockchain/
• aws-sigv4-proxy

Once I figure that out and have the miner running on mainnet for a bit I'll finish up the document with the specifications for a miner and follower. Once the final Signer PRs are published I'll test a signer node and add that to the document - the resources required for setup are going to be very similar across all node types, it's just the parameters that'll change.

[AshtonStephens]

Ran into some configuration issues getting the Stacks prometheus metrics to be picked up by CloudWatch. The documentation and binaries were made with Amazon Linux in mind, so I think I'll move the whole setup to use AL2 instead of debian.

I've been able to setup permissions such that the host can connect to AMB, but I'm running into some issues with the proxy failing here: https://github.com/awslabs/aws-sigv4-proxy/blob/master/handler/proxy_client.go#L183

I need to look more into the configurations tomorrow to see what I'm doing wrong.

Once we have the cloudwatch metrics and the proxy to Bitcoin working all that's left is consuming Secrets from Secrets Manager which looks straightforward. Then we'll know everything it takes to automate the setup

[AshtonStephens]

Tl;dr; I'm not going to add a Miner configuration in the first release.

I was able to get rpc to access Amazon's hosted Bitcoin network through with this aws-sigv4-proxy configuration.

sudo docker run --rm -ti \
  -e 'AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID' \
  -e 'AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY' \
  -p 8080:8080 \
  aws-sigv4-proxy \
	--host mainnet.bitcoin.managedblockchain.us-east-1.amazonaws.com \
	--name managedblockchain \
	--region us-east-1

You can test it with this command:

curl \
  --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getbestblockhash", "params": [] }' \
  -H 'content-type: text/plain;' \
  http://127.0.0.1:8080

As it turns out, the majority of the communication between Stacks nodes and Bitcoin nodes is through p2p; the Stacks node only uses the rpc port when it's mining to send transaction data for the block-commit.

In addition to Bitcoin rpc access, the Stacks miner needs access to the UTXOs for the miner wallet and it cannot get those purely from the rpc endpoint. I have to dig deeper on how the Stacks node queries for miner wallet UTXOs, but right now the usual path is to host a bitcoind node locally and manually connect the bitcoind node to the miner wallet.

AWS has an API called AMB Query which has a few API calls that should allow us to get the UTXO information. I think ListTransactions is likely the right API call, but we'd need to wrap that API call such that the Stacks node thinks it's still interacting with a bitcoind node. Right now, it's just not worth it.

---

The next step is to get a Follower working with prometheus metrics on AL2. Amazon's Bitcoin endpoint doesn't serve p2p data so we'll likely need to connect to some arbitrary trusted Bitcoin node.

Next week the Signer happy path should be finished; which means we will be able to manually set that up as well.

One note on p2p query packet sending: a mock mining node sends around 613 packets a second.

[AshtonStephens]

I've sent the follower / signer documentation for an initial review from @wileyj and @hstove and anticipate needing to make minor changes.

Against what I wrote above, I want to spend just a little more time trying to solve the UTXO issue for the miner because that's the ONLY weak link in the automatic miner setup. Getting miners to automatically spin up will make testing a lot easier on testnet.

Once we can get the UTXOs reporting from whatever bitcoin endpoint the Stacks Miner node is connect to we can have the miner setup automated.

[AshtonStephens]

Going slightly out of order - I've had to go through the full automation sequence of the follower node to really understand what the different areas are to tweak in the CDK. Once I have that entirely automated I can flesh out the document to its final form including the signer. There aren't many design choices in this initial part and the ones that existed were documented and reviewed as part of the original doc.

[AshtonStephens]

Finished automating the follower, I'll clean it up and put out a draft PR tomorrow.

[AshtonStephens]

Opened a PR for a single node into a feature fork of the aws-samples/aws-blockchain-node-runners repository:

• https://github.com/stacks-network/aws-blockchain-node-runners/pull/1

[AshtonStephens]

Design document: https://docs.google.com/document/d/1yIvmJk--r5je7hc9JewzRM5C1G7_T33apkCRHeOw0hA/edit

[AshtonStephens]

Closing because this issue was completed a while back but with different entities. https://aws-samples.github.io/aws-blockchain-node-runners/docs/Blueprints/Stacks