[QT-645] Restructure dev tools

[ryancragun]

Our previous process of installing internal and external developer tools was scattered and inconsistent. Some tools were installed via go generate -tags tools ./tools/..., others via various make targets, and some in Github Actions workflows. This process led to some undesireable side effects:

• The modules of Go dev tools were included with those of the Vault project. This leads to us having to manage our own modules with those of the developer tools.
• Due to Github cache constraints we combine our build and test Go module caches together, but having our developer tools as deps in our module results in a larger cache that has to be uploaded when a go.mod changes. This same cache is then downloaded on every CI and Build runner. When we remove the developer tools that were in the module we reduce our expanded module cache by ~300MB.
• Not all of our developer tools were included in our modules. This leads to us downloading Go modules on every CI and Build runner in order to build our developer tools.
• Building our developer tools from source in CI is slow. It is often preferable to install pre-built tool binaries.

Instead, I've refactored how we define internal and external build tools in our Makefile and added several new targets to handle both building the developer tools locally for development and verifying that they are available. This allows for an easy developer bootstrap while also supporting installation of many of the external developer tools from pre-build binaries in CI. This reduces our network IO and run time across nearly all of our actions runners.

While working on this I caught and resolved a few unrelated issue:

• Both our Go and Proto format checks we're being run incorrectly. In CI they we're writing changes but not failing if changes were detected.
• Our Git hooks couldn't handle removing a Go file without failing. I moved the diff check into the new Go helper and updated it to handle removing files.

Changes

• Refactor our internal and external tool installation and verification into a tools.sh helper.
• Combine more complex Go verification into scripts/go-helper.sh and utilize it in the Makefile and git commit hooks.
• Add Makefile targets for executing our various tools.sh helpers.
• Update our existing make targets to use new tool targets.
• Normalize our various scripts and targets output to have a consistent format.
• In CI, install our external dependencies as binaries wherever possible. When not possible we'll build them from scratch but not mess with the shared module cache.
• QT-641 Remove our external build tools from our project Go modules.
• QT-641 Remove extraneous go list's from our set-up-to composite action.

Signed-off-by: Ryan Cragun [email protected]

[github-actions[bot]]

CI Results: All Go tests succeeded! :white_check_mark:

[github-actions[bot]]

Build Results: All builds succeeded! :white_check_mark:

[ryancragun]

One thing I did not implement in this PR that I thought about doing was installing all of these tools into a directory that we could cache. The idea being that restoring a cache of our tools would be the absolute fastest options in CI, but it would require some of our little but valuable space. I was hoping to get this merged through the pipeline and then observe our cache history for a week to two to see if we could safely start caching the tools without risk of ejecting relevant module and/or timing data.