Deploying Vault Enterprise in a private datacenter

Overview

This section details the steps to create a Vault cluster manually in a private datacenter. This guide assumes that you have already read the Detailed Design section of this guide and have a basic understanding of the Vault architecture and the steps required to deploy Vault in a private datacenter.

Throughout this section we will use the following names and IP addresses for the Vault servers:

These names and IP addresses are used for illustrative purposes only. Use the names and IP addresses that are appropriate for your environment.

Prepare

Create or obtain the files that you will later deploy onto the Vault servers. The Vault binaries are pulled via curl command during the Installation phase.

TLS

Create an X.509 certificate that will be installed onto each of the Vault servers. Refer to your organization's process on creating a new certificate that matches the DNS record you intend to direct users to when accessing Vault - in this case, the DNS record pointing at the load balancer: vault.domain.

Three files will be needed:

1. The certificate (vault-public.pub).
2. The certificate's private key (vault-private.key).
3. The bundle file from the certificate authority used to vend the certificate (ca.pub).

License

Obtain your active Vault Enterprise License file. If you do not have this file, contact your HashiCorp account team.

Server configs

Using what you have learned from previous sections of this guide, create 6 configuration files, one for each server. The configuration for each server will be slightly different, depending on which redundancy zone it lives in, and whether it is a voting or a non-voting node.

Voting node example file

# config.hcl for vault1 (voter node example)
api_addr = "https://10.0.0.1:8200"
cluster_addr = "https://10.0.0.1:8201"
ui = true
 
license_path = "/etc/vault.d/vault.hclic"
 
listener "tcp" {
  address     = "10.0.0.1:8200"
  cluster_address = “10.0.0.1:8201”
  tls_client_ca_file = "/etc/vault.d/ca.pub"
  tls_cert_file = "/etc/vault.d/vault-public.pub"
  tls_key_file  = "/etc/vault.d/vault-private.key"
}
 
storage "raft" {
  path    = "/var/lib/vault/data"
  node_id = "vault1"
  autopilot_redundancy_zone = "zone1"
 
  retry_join {
    leader_api_addr     = "https://10.0.0.1:8200"
    leader_ca_cert_file = "/etc/vault.d/ca.pub"
    leader_client_cert_file = "/etc/vault.d/vault-public.pub"
    leader_client_key_file  = "/etc/vault.d/vault-private.key"
    leader_tls_servername = “vault.domain”
}
 
  retry_join {
    leader_api_addr     = "https://10.0.0.2:8200"
    leader_ca_cert_file = "/etc/vault.d/ca.pub"
    leader_client_cert_file = "/etc/vault.d/vault-public.pub"
    leader_client_key_file  = "/etc/vault.d/vault-private.key"
    leader_tls_servername = “vault.domain”
  }
 
  retry_join {
    leader_api_addr     = "https://10.0.0.3:8200"
    leader_ca_cert_file = "/etc/vault.d/ca.pub"
    leader_client_cert_file = "/etc/vault.d/vault-public.pub"
    leader_client_key_file  = "/etc/vault.d/vault-private.key"
    leader_tls_servername = “vault.domain”
  }
 
  retry_join {
    leader_api_addr     = "https://10.0.0.4:8200"
    leader_ca_cert_file = "/etc/vault.d/ca.pub"
    leader_client_cert_file = "/etc/vault.d/vault-public.pub"
    leader_client_key_file  = "/etc/vault.d/vault-private.key"
    leader_tls_servername = “vault.domain”
  }
 
  retry_join {
    leader_api_addr     = "https://10.0.0.5:8200"
    leader_ca_cert_file = "/etc/vault.d/ca.pub"
    leader_client_cert_file = "/etc/vault.d/vault-public.pub"
    leader_client_key_file  = "/etc/vault.d/vault-private.key"
    leader_tls_servername = “vault.domain”
  }
 
  retry_join {
    leader_api_addr     = "https://10.0.0.6:8200"
    leader_ca_cert_file = "/etc/vault.d/ca.pub"
    leader_client_cert_file = "/etc/vault.d/vault-public.pub"
    leader_client_key_file  = "/etc/vault.d/vault-private.key"
    leader_tls_servername = “vault.domain”
  }
}
 
seal "awskms" {
  region     = "us-west-2"
  kms_key_id = "abcd1234-a123-456a-a12b-a123b4cd56ef"
}
 
telemetry {
  disable_hostname = true
  prometheus_retention_time = "12h"
}

Non-voting node example file

# config.hcl for vault4 (non-voter node example)
 
api_addr = "http://10.0.0.4:8200"
cluster_addr = "https://10.0.0.4:8201"
ui = true
 
license_path = "/etc/vault.d/vault.hclic"
 
storage "raft" {
  path    = "/var/lib/vault/data"
  node_id = "vault4"
  autopilot_redundancy_zone = "zone1"
}
 
listener "tcp" {
  address     = "10.0.0.4:8200"
  cluster_address = “10.0.0.4:8201”
  tls_client_ca_file = "/etc/vault.d/ca.pub"
  tls_cert_file = "/etc/vault.d/vault-public.pub"
  tls_key_file  = "/etc/vault.d/vault-private.key"
 
  retry_join_as_non_voter = true
 
  retry_join {
    leader_api_addr     = "https://10.0.0.1:8200"
    leader_ca_cert_file = "/etc/vault.d/ca.pub"
    leader_client_cert_file = "/etc/vault.d/vault-public.pub"
    leader_client_key_file  = "/etc/vault.d/vault-private.key"
    leader_tls_servername = “vault.domain”
  }
 
  retry_join {
    leader_api_addr     = "https://10.0.0.2:8200"
    leader_ca_cert_file = "/etc/vault.d/ca.pub"
    leader_client_cert_file = "/etc/vault.d/vault-public.pub"
    leader_client_key_file  = "/etc/vault.d/vault-private.key"
    leader_tls_servername = “vault.domain”
  }
 
  retry_join {
    leader_api_addr     = "https://10.0.0.3:8200"
    leader_ca_cert_file = "/etc/vault.d/ca.pub"
    leader_client_cert_file = "/etc/vault.d/vault-public.pub"
    leader_client_key_file  = "/etc/vault.d/vault-private.key"
    leader_tls_servername = “vault.domain”
  }
 
  retry_join {
    leader_api_addr     = "https://10.0.0.5:8200"
    leader_ca_cert_file = "/etc/vault.d/ca.pub"
    leader_client_cert_file = "/etc/vault.d/vault-public.pub"
    leader_client_key_file  = "/etc/vault.d/vault-private.key"
    leader_tls_servername = “vault.domain”
  }
 
  retry_join {
    leader_api_addr     = "https://10.0.0.6:8200"
    leader_ca_cert_file = "/etc/vault.d/ca.pub"
    leader_client_cert_file = "/etc/vault.d/vault-public.pub"
    leader_client_key_file  = "/etc/vault.d/vault-private.key"
    leader_tls_servername = “vault.domain”
  }
}
 
seal "awskms" {
  region     = "us-west-2"
  kms_key_id = "abcd1234-a123-456a-a12b-a123b4cd56ef"
}
 
telemetry {
  disable_hostname = true
  prometheus_retention_time = "12h"
}

Set aside these files for later in the installation process.

Build

Servers

1. Identify the availability zones within your datacenter where your Vault servers will live. For the rest of this document, we will refer to these as zone1, zone2, etc.
2. Build the servers. Ensure there are 2 servers per each of the 3 redundancy zones, for a total of 6 servers. For the rest of this document, we will refer to these servers as vault1, vault2, etc.
3. Ensure you can log in to each server as the root user (or a user with sudo privileges) via SSH or equivalent.

Load balancer

Configure your load balancer with a new frontend for Vault that will route incoming traffic on TCP port 8200 to the Vault servers on TCP port 8200. Refer to the Detailed Design section of the guide for more information on how to configure the load balancer.

Snapshot storage

Ensure that your snapshot storage location is created and that the Vault service can successfully write to it. If you will be using a remote network filesystem, ensure that it is mounted at the correct location on all Vault servers.

Configuration of Vault snapshots can be found in Vault: Operating Guide for Adoption.

Install Vault

Perform these steps on all Vault servers:

1. Create a vault user. Create directories for Vault configuration owned by this user.

$ useradd --system --user-group --shell /bin/false vault
$ mkdir -p /etc/vault /var/lib/vault
$ chown -R vault:vault /etc/vault /var/lib/vault /usr/local/bin/vault

1. Download the Vault Enterprise package from Hashicorp. Unzip the package and move the vault binary to a shared PATH location such as /usr/local/bin, owned by the vault user.

$ curl -O https://releases.hashicorp.com/vault/1.15.0+ent/vault_1.15.0+ent_linux_amd64.zip
$ unzip vault_1.15.0+ent_linux_amd64.zip
$ mv vault /usr/local/bin/
$ chown vault:vault /usr/local/bin/vault

1. Copy the files created in the prepare section above (license file, certificate files, and relevant config file) to this server at /etc/vault. Ensure that the files are owned by the vault user and that the permissions are set to 640.

$ chown vault:vault /etc/vault.d/*
$ chmod 640 /etc/vault.d/*
$ ls -l /etc/vault.d/

-rw-r---- 1 vault vault 0 Sep 22 12:45 ca.pub
-rw-r---- 1 vault vault 0 Sep 22 12:45 config.hcl
-rw-r---- 1 vault vault 0 Sep 22 12:45 vault.hclic
-rw-r---- 1 vault vault 0 Sep 22 12:45 vault-public.pub
-rw-r---- 1 vault vault 0 Sep 22 12:45 vault-private.key

1. Create a systemd unit file for the Vault service, then load it into systemd. Note that the ExecStart line runs the vault binary pointing to your config.hcl file:

$ cat << EOF >> /etc/systemd/system/vault.service
[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://developer.hashicorp.com/vault/docs
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/config.hcl
StartLimitIntervalSec=60
StartLimitBurst=3

[Service]
Type=notify
EnvironmentFile=/etc/vault.d/vault.env
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/vault server -config=/etc/vault.d/config.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity
LimitCORE=0

[Install]
WantedBy=multi-user.target
EOF


$ systemctl daemon-reload
$ systemctl enable vault

1. Start the Vault service.

$ systemctl start vault

Initialize

Create GPG public keys

For each individual who will be receiving a key share, create their own unique GPG public key:

$ cat > alice_key.conf << EOF
%echo Generating a basic OpenPGP key for Alice
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Name-Real: Alice
Name-Comment: Alice is a Vault PGP user
Name-Email: alice@example.com
Expire-Date: 1
Passphrase: recede-yard-unwilling-shrouded
%commit
%echo done
EOF

$ gpg --full-gen-key --batch alice_key.conf
...
$ gpg --output alice_key.pub --export alice@example.com

Initialize Vault

Pass in the GPG public key from each participating individual to initialize Vault with the API, CLI, or UI. Once Vault initializes, it returns the key shares, each one encrypted with the GPG public keys which you passed in at initialization time.

$ vault operator init \
  -pgp-keys “alice_key.pub,bob_key.pub,carol_key.pub,dan_key.pub,frank_key.pub" \
  -root-token-pgp-key "alice_key.pub"


Unseal Key 1: wcFMA7Np...JCbg=
Unseal Key 2: wcFMAw4D...5LY4=
Unseal Key 3: wcFMA8d6...G8r0=
Unseal Key 4: wcFMA4ly5...6ELY=
Unseal Key 5: wcFMA0eb...3hRg=


Initial Root Token: wcFMA7Np…2mGO


Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.


Vault does not store the generated root key (previously known as master key).
Without at least 3 keys to reconstruct the root key, Vault will remain permanently sealed!


It is possible to generate new unseal keys, provided you have a quorum of existing unseal keys shares. See "vault operator rekey" for more information.

1. Record the root token value from this output. You will need it in order to log into Vault initially.
2. Each individual should now record their unique unseal key and store it safely. This is their key share, which is now encrypted and base64-encoded.

Unseal Vault

To unseal Vault using encrypted key shares, each individual will need to both base64-decode and decrypt their unseal key before sending it to the Vault server. When decrypting with GPG each individual will also need to pass the unique passphrase that was supplied during the creation of their own GPG key:

$ cat encoded-encrypted-keyshare1 | base64 --decode > encrypted-keyshare1
$ gpg --passphrase recede-yard-unwilling-shrouded --decrypt encrypted-keyshare1 > keyshare1
$ vault operator unseal keyshare1


gpg: encrypted with 4096-bit RSA key, ID 91002A8F909C6714, created 2023-04-17
      "Alice (Alice is a Vault PGP user) <alice@example.com>"
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
…

Repeat this process until a sufficient number of keyshares have been supplied to Vault. At this point the Vault server is now unsealed. The output of the final unseal command should look something like the below.

gpg: encrypted with 4096-bit RSA key, ID 6ED85F446B6FAA27, created 2023-04-17
      "Dan (Dan is a Vault PGP user) <dan@example.com>"
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
…

This concludes the key ceremony.

Form the cluster

You should now have a functional, initialized and unsealed cluster, which contains one active node: vault1. You can confirm this by setting the VAULT_TOKEN environment variable to the root token generated during the init step and then running:

$ vault operator raft list-peers

You should now see that the cluster is comprised of six nodes: one active node, and five standby nodes. To view the voter/non-voter status of each node, you can run:

$ vault operator raft autopilot state

Congratulations, you now have a running cluster! You can now test that you are able to log into Vault:

$ export VAULT_TOKEN=<root_token_value>
$ vault login -no-store $VAULT_TOKEN