Nodes in Vault clusters (and Consul clusters if being used as a storage backend) should be spread accross two or more failure domains known as Availability zones. The loss of a single Availability zone should not result result in a loss of service.

If you are deploying your Vault nodes on virtual machines, It is reccomended to build re-usable VM images that can be used to create cluster nodes in an immutable way. Tools like Hashicorp Packer are designed to help build repeatable machine images for most virtualised and cloud platform. Machine images should be versioned and should follow a release cycle as new images are produced.

If you are using the Enterprise version of Vault, you can enable replication between two or more Vault clusters in different geographical regions for added protection is Disaster Recovery scenarios. Replication can be configured in Disaster Recovery Mode or Performance Replication mode. If you are planing on using Replication, you need to provision infrastructure in an alternative region, with nodes spread accross multiple Availability Zones. For more information about the Enterprise Replication feature, see the official documentation.

Vault will likely contain business critical secrets which makes it a prime target for malicious actors. Access to vault to should be restricted to your private networks and not be accessible on the internet. The Use of Virtual Private Networks is a commonly used approach to allow access to Vault from unknown networks

Ensure Hardware servers and Virtual Machines have been appropriately resources in accordance with the Deployment System Requirements

Ensure that vault servers have a secondary disk attached to them. This will help with Audit Device Fault tolerance

It is a recommended pattern to use HashiCorp Consul's Key/Value store as the storage backend. The reccomended cluster size for consul is 5 nodes. This cluster size allows for fault tolerance whilst performing maintenence on a a node

Members of the Consul clusters use a gossip protocol to communicate with eachother and hold leadership elections. This network traffic should be encrypted to minimise security risks. You can read more about consul encryption here.

The path that Vault uses in Consul's key/value storage to store it's encrypted data should be protected using Consul's ACL system. Once configured, the Management ACL token should be revoked. You can read more about configuring Consul's ACL system here.

As Consul is being used as a data store that Vault uses, it should be considered a stateful service, and as such, should have a backup strategy. Consul snapshot, in addition to disk backups should be implemented on a regular schedule. For more information about consul snapshot, click here

Vault should not talk directly to Consul backend as this introduces an increased attack vector. Instead, Consul should be installed on the Vault servers and configured in client mode. The clients will facilitate the communication between Vault and Consul.

If using a 5 node consul cluster, you can choose to enable the UI; however, it is recommended that the UI is enabled on two nodes only.

Vault’s communications should be encrypted end-to-end with TLS and this should not be terminated at the Load balancer layer. The load balancer should also use the same encryption to communicate with Vault

With TLS configured, all traffic going via HTTPS will be encrypted; however, we need to ensure that there are no connections to vault via HTTP. The Load balancer should be configured to redirect all HTTP traffic to HTTPS.

Load balancer health probes can be used to ensure that traffic is only routed to a healthy leader node. Configure routing rules according to these response codes

Vault telemetry should be configured in the telemetry stanza within the Vault config file. This will enable monitoring and alerting with a wide range of open source tools (Telegraf and prometheus)

Consul telemetry should be configured in the telemetry stanza within the consul config file. This will enable monitoring and alerting with a wide range of open source tools (Telegraf and prometheus)

Monitoring system of your choice is configured to monitor and alert on vault application metric thresholds as per the best practice guidance of Hashicorp.

Monitoring system of your choice is configured to monitor and alert on infrastructure metric thresholds as per the best practice guidance of Hashicorp.

Monitoring system of your choice is configured to monitor and alert on consul application metric thresholds as per the best practice guidance of Hashicorp.

Using a Dashboard tool a of your choice, create a monitoring dashboard for operations staff to easily identify any issues that may be occurring.

Code written to deploy the infrastructure for Consul and Vault. Terrafrom is an appropriate tool for this task. Virtual Machine images created from code for Consul and Vault. Packer is a good choice of tool for this. All Virtual infrastructure should be deployed and managed using an Infrastructure as code tool

Vault Platform configuration should be described in code using a tool like Terrafrom. Configuration such as Auth Methods, Secrets Engines, Audit Devices and Policies should all be configured using code

All Infrastructure code and application code should be stored separate source control repositories and be placed under version control. An appropriate branching strategy should be implemented and documented in the README file.

Repository files should have code owners assigned to them to control who can approve Pull Requests that will be merged into the Master branch.

Configure the minimum number of Pull Request approvers, restrictions on Pull Request Authors approving their own requests and any other rules that your organisation’s security standards require for Integrity.

Code deployments should be automated using deployment pipelines. Where possible, the pipeline should be written as code and stored under version control with the code

Create development and staging environments for Vault. Staging Environment should be identical to production, with the only divergence being, when pre-production changes are implemented for final testing prior to being deployed to production.

Implement and document naming and coding standards. Naming standards for Namespaces, Policies, Vault Roles, secrets keys and AppRoles. Coding standards where applicable for variable names and function names.

A suite of automated integration tests written to be run either during the deployment pipeline or as a pre-check on your chosen VCS required to pass before a Pull Request can be merged.

Vault logs all requests and responses to requests. If Vault is unable to log requests and responses to these requests, it will immediately seize operations. To provide redundancy, each vault node should have 2 file audit devices enabled on separate volumes on separate disks.

Enable and configure log rotation on the audit files to ensure the disks do not fill up and cause a vault outage.

Where appropriate, configure an existing identity provider (or multiple if required) as an authentication method in Vault

Identify and enable the required secrets engines for your business and technical use cases

Ensure that Version 2 of the KV secrets engine is used to enable secrets versioning

Create default policies that all user entities will inherit according to your business security model. This could be list permissions on a particular KV path for example.

Create a mapping for default policies to ensure all user entities inherit these policies.

Using the Identity Secrets engine, create aliases to attach vault logins via different auth methods to a single entity to ensure the correct policies are inherited and to make the logging data easier to mine

Map you KV path design to the way your organisation works or product groupings.

Meta Approles are a mechanism that allow an application or service to read the secret id of an app role without exposing this to application developers.

Enable end-to-end encryption using TLS certificates. Vault agents should also use TLS certificates

Enable end-to-end encryption on consul cluster and agent. More information can be found here.

Enable and config SELinux / app amour depending on your operating system to create sandboxed contexts to reduce blast radius if even the system is compromised.

By default, Vault uses port 8200 and 8201. Change the port to a non-standard port to provide extra hardening

By default, Consul uses port 8500 and 8501. Change the port to a non-standard port to provide extra hardening

Once initial set-up of Vault cluster has been completed, the root token should be revoked.

Once the initial set-up of the Consul ACL system has been completed, the management token should be revoked.

Using firewalld or IP Tables, configure these firewalls to limit port access to the vault and consul servers.

Interaction with Vault is done via the API, even when using the CLI. As such, there is no reason to have to SSH on to a vault server (if it’s a virtual machine) so SSH should be disabled to mitigate the risk of unauthorised access to the server.

Add a seal stanza to the Vault config file to reduce operational burden on operators. For more information check the auto-unseal documentation here

Use PGP or Keybase to add an extra layer of security to the distribution of unseal/recovery keys. For more details, see the official documentation here

Practice restoring your Vault platform from a Consul snapshot. Your backup strategy isn’t complete until you have tested this.

Practice building and replacing a node in the vault and consul clusters with zero downtime.

Practice upgrading Vault binaries to newer versions with zero downtime.

Practice upgrading Consul binaries to newer versions with zero downtime.

Consuct load testing to ensure your infrastructure compute resources are sufficient for the load you are expecting. There are projects like wrk That can assist with generating traffic.

Ensure unseal/recovery key holders are documented on a Wiki and this document is kept up-to-date

Choose a platform or broker that your business trusts and use this for secure injection of initial secrets. Examples are using Azure as a trusted platform or using Jenkins as a trusted broker. Each organisation will differ with regards to what they trust so this should be a business driven decision.

Document an implement an OS patching strategy, whether it’s updating VM images and replacing VMs with up-to-date images or whether its a controlled direct access update by an operator.

Logs should be streamed to a central data warehouse as log rotation on the servers should be enabled and logs will be lost locally. A platform like splunk is ideal for this use case. There are other viable options available.

Decide the value that the log data should provide and write some scripts to extract this value from the data. Scripts can be written in python. Models can also be produced to predict future loads based on existing data sets. This kind of insight can be useful for planning.

Some events should generate some kind of alert, for example, a root token being generated should be flagged and alerted on. Ensure these events have alerts configured for them.

Conduct a threat modelling exercise using a framework of your organisations choosing and ensure you have documented and mitigated against all identified threats.