solana icon indicating copy to clipboard operation
solana copied to clipboard

Add Kubernetes/Monogon support for test deployments

Open gregcusack opened this issue 1 year ago • 1 comments

Problem

  1. GCE is expensive.
  2. We need to be able to run test deployments with 1000s of validators. We have Monogon servers, so let's put Solana validators in kubernetes pods and spin up a bunch at a time.

Summary of Changes

Current functionality includes:

  • Build a validator from either local git hash or pull release version from solana releases
  • Build docker image and push to docker registry
  • Generate faucet, bootstrap, and other validator accounts
  • Generate genesis with bootstrap account (including spl-genesis-args).
  • Pull docker images, build kubernetes controller and deploy 1 validator per pod. 1 pod per replicaset. 1 service per replicaset
  • Mounts genesis (as configmap) and respective accounts (as secrets) into each validator pod
  • Bootstrap validator starts up
  • Validators wait for bootstrap to start before deploying and connecting to bootstrap.
  • Can exec into bootstrap pod and verify validators are all connected with solana -ul gossip and solana -ul validators
  • Metrics Enabled
  • Client support

See solana/k8s-cluster/README.md for a tutorial on how to run for yourself.

WIP:

  • adding in additional validator configuration flags. that is coming.....

TODO:

  • blockstreamer Fixes #

gregcusack avatar Oct 06 '23 23:10 gregcusack

Codecov Report

Attention: Patch coverage is 4.96454% with 1206 lines in your changes are missing coverage. Please review.

Project coverage is 81.2%. Comparing base (b814497) to head (e9c9eef). Report is 354 commits behind head on master.

:exclamation: Current head e9c9eef differs from pull request most recent head eb60e2e. Consider uploading reports for the commit eb60e2e to get more accurate results

Additional details and impacted files
@@            Coverage Diff            @@
##           master   #33574     +/-   ##
=========================================
- Coverage    81.8%    81.2%   -0.6%     
=========================================
  Files         824      831      +7     
  Lines      222394   223911   +1517     
=========================================
+ Hits       181957   182027     +70     
- Misses      40437    41884   +1447     

codecov[bot] avatar Nov 06 '23 20:11 codecov[bot]

This repository is no longer in use. Please re-open this pull request in the agave repo: https://github.com/anza-xyz/agave

willhickey avatar Mar 03 '24 04:03 willhickey