Hari Sekhon - DevOps Golang Tools

git.io/go-tools

DevOps Golang Tools

All programs have --help to list the available options.

See Also:

• DevOps Bash Tools
• DevOps Python Tools
• DevOps Perl Tools
• Advanced Nagios Plugins Collection

repos which contain hundreds of Cloud, Big Data, NoSQL, Web, Linux and other DevOps infrastructure tools.

Hari Sekhon

Cloud & Big Data Contractor, United Kingdom

https://www.linkedin.com/in/HariSekhon

(you're welcome to connect with me on LinkedIn)

Make sure you run make update if updating and not just git pull as you will often need the latest library submodule and possibly new upstream libraries

Quick Start

Ready to run Docker image

All programs and their pre-compiled dependencies can be found ready to run on DockerHub.

List all programs:

docker run harisekhon/go-tools

Run any given program:

docker run harisekhon/go-tools <program> <args>

Automated Build from source

installs git, make, pulls the repo and build the binaries:

curl -L https://git.io/go-bootstrap | sh

or manually:

git clone https://github.com/HariSekhon/DevOps-Golang-tools go-tools
cd go-tools
make

The make command automates building the go binaries which can then be copied around to other systems of the same family, eg. Linux or Mac.

Alternatively there is shebang magic which means each .go program can be called directly like a script and it'll runtime compile and execute instantly like a scripted language. This is a neat trick for quick usage and testing built on go run, but for frequent use the compiled binaries are usually the way to go.

Detailed Build Instructions are available further down.

Usage

All programs come with a --help switch which includes a program description and the list of command line options.

Environment variables are supported for convenience and also to hide credentials from being exposed in the process list eg. $PASSWORD, $TRAVIS_TOKEN. These are indicated in the --help descriptions in brackets next to each option and often have more specific overrides with higher precedence eg. $AMBARI_HOST, $HBASE_HOST take priority over $HOST.

DevOps Golang Tools - Inventory

• Linux:
  • uniq2.go - like uniq but you don't have to sort first and it preserves the ordering
  • httpfirst.go - returns the first http/https url address argument to respond (fastest multi-threaded reply using go channels). More sophisticated version is find_active_server.py in the DevOps Python tools repo which can handle multi-master clusters, tcp sockets, regex etc.
  • colors.go - prints a table of terminal colors and their escape codes for doing fancy shell stuff
  • welcome.go - cool spinning welcome message greeting your username and showing last login time and user to put in your shell's .profile (there are also Python and Perl versions in my DevOps Python Tools and DevOps Perl Tools repos)

Detailed Build Instructions

Manual Setup

Enter the go-tools directory and run git submodule init and git submodule update to fetch my library repo:

git clone https://github.com/HariSekhon/DevOps-Golang-tools go-tools
cd go-tools
git submodule update --init
./compile.sh

Mac OS X

The automated build also works on Mac OS X but you'll need to install Apple XCode (on recent Macs just typing git is enough to trigger Xcode install).

I also recommend you get HomeBrew to install other useful tools and libraries you may need like OpenSSL for development headers and tools such as wget (these are installed automatically if Homebrew is detected on Mac OS X):

bash-tools/install/install_homebrew.sh

Updating

Run make update. This will git pull and then git submodule update which is necessary to pick up corresponding library updates.

Contributions

Patches, improvements and even general feedback are welcome in the form of GitHub pull requests and issue tickets.

Related Repositories

• DevOps Bash Tools - 1000+ DevOps Bash Scripts, Advanced .bashrc, .vimrc, .screenrc, .tmux.conf, .gitconfig, CI configs & Utility Code Library - AWS, GCP, Kubernetes, Docker, Kafka, Hadoop, SQL, BigQuery, Hive, Impala, PostgreSQL, MySQL, LDAP, DockerHub, Jenkins, Spotify API & MP3 tools, Git tricks, GitHub API, GitLab API, BitBucket API, Code & build linting, package management for Linux / Mac / Python / Perl / Ruby / NodeJS / Golang, and lots more random goodies

• SQL Scripts - 100+ SQL Scripts - PostgreSQL, MySQL, AWS Athena, Google BigQuery

• DevOps Python Tools - 80+ DevOps CLI tools for AWS, GCP, Hadoop, HBase, Spark, Log Anonymizer, Ambari Blueprints, AWS CloudFormation, Linux, Docker, Spark Data Converters & Validators (Avro / Parquet / JSON / CSV / INI / XML / YAML), Elasticsearch, Solr, Travis CI, Pig, IPython

• DevOps Perl Tools - 25+ DevOps CLI tools for Hadoop, HDFS, Hive, Solr/SolrCloud CLI, Log Anonymizer, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Dockerfiles & SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Ambari FreeIPA Kerberos, Datameer, Linux...

• The Advanced Nagios Plugins Collection - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)

• Nagios Plugin Kafka - Kafka API pub/sub Nagios Plugin written in Scala with Kerberos support

• HAProxy Configs - 80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, Cloudera, Hortonworks, MapR, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, ZooKeeper, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, SSH, RabbitMQ, Redis, Riak, Rancher etc.

• Templates - dozens of Code & Config templates - AWS, GCP, Docker, Jenkins, Terraform, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions Workflows, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml, M4 etc.

• Kubernetes configs - Kubernetes YAML configs - Best Practices, Tips & Tricks are baked right into the templates for future deployments

• Terraform - Terraform templates for AWS / GCP / Azure / GitHub management

• Jenkins - Advanced Jenkinsfile & Jenkins Groovy Shared Library

• GitHub-Actions - GitHub Actions master template & GitHub Actions Shared Workflows library

• Dockerfiles - 50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Mesos, Consul, Riak, OpenTSDB, Jython, Advanced Nagios Plugins & DevOps Tools repos on Alpine, CentOS, Debian, Fedora, Ubuntu, Superset, H2O, Serf, Alluxio / Tachyon, FakeS3

• HashiCorp Packer templates - Linux automated bare-metal installs and portable virtual machines OVA format appliances using HashiCorp Packer, Redhat Kickstart, Debian Preseed and Ubuntu AutoInstaller / Cloud-Init

• PyLib - Python library

• Perl Lib - Perl library

• Diagrams-as-Code - Cloud & Open Source architecture diagrams with Python & D2 source code provided - automatically regenerated via GitHub Actions CI/CD - AWS, GCP, Kubernetes, Jenkins, ArgoCD, Traefik, Kong API Gateway, Nginx, Redis, PostgreSQL, Kafka, Spark, web farms, event processing...

Stargazers over time

git.io/go-tools