Design: Rethink about the Netgoal implementation

[yaovi-a]

Context

Netgoal is tool used to create network files with 30M or more accounts, and it also funds those accounts so that they can be used. It allows us to spin up a network with pre-generated accounts. There are currently two challenges here. Ping pong can't access these accounts because it does not have access to them -- private keys. Secondly, netgoal does not take the most optimal approach when creating these network files.

Objective 

Redesign how netgoal generates cloudspec.config

How does netgoal generate aws network today:

1. Netgoal uses the network, host template and topology json files is used to generate the cloudspec.config
2. Algonet then uses a string templating language to generate the TF network files based on the cloudspec.config
3. Algonet does the terraform apply using some tfvar files that are autogenerated based on the cloudspec and cli arguements. The AWS infrastructure is then created

How we rather do it:

All of these templating steps could be done using terraform rather than these various custom templating languages we've developed. We can come up with a tfvar schema with feature parity to what we have today, write tf modules to implement them, and store tfvars files in our recipe folders for all of the cloud configurations. This way recipe folders will just have algorand network configs + tf vars files, which should improve usability/readability

Value Add 

UX Removing race condition when creating cloudspec.config.

1. This can cause perf test to fail -- when algonet destroy is used, in some cases we have to run the command again.
2. Reduce the chances of having resources running and left unchecked.

Tech Debt Reducing complexity: Focus netgoal on algorand network vs cloud infrastructure

Design notes 

Netgoal redesign:

• Use netgoal for constructing network files and use TF to create the network
• Utilize ansible and TerraForm

Note: consider including how security groups are created to avoid race conditions

Success criteria

Provide a design documentation for how netgoal generates cloudspec.config Create follow up implementation