Manual install (private cloud)

Our Terraform-based Terraform Enterprise deployment modules focus on public cloud deployments because public CSPs offer a predictable set of services. Even though the overwhelming majority of our private cloud customers use VMware, there is still a bespoke approach to deployment, instead using a wide variety of additional configuration management technologies and deployment approaches to manage software installation, so an opinionated private cloud deployment Terraform module would be unlikely to be sufficiently useful for all private cloud customers.

This section thus provides recommendations and best practices for scaled, enterprise private cloud customers who need to apply their organisation-standard tooling to the deployment of Terraform Enterprise.

If you are deploying Terraform Enterprise to a public cloud, please refer to the Terraform-based Terraform Enterprise installation section for instructions.

Deployment topology

As listed in the architecture section, HashiCorp recommends the Active/Active deployment topology for customers seeking to guarantee high availability, resilience and scale.

As an automated deployment is a requirement of Active/Active, engineering to provide this should be a headline planning item for the project manager.

Component considerations

This section discusses a series of considerations specific to Active/Active private cloud installations irrespective of whether Docker or Kubernetes is used to manage Terraform Enterprise.

Redis

As a result of there being multiple concurrent instance nodes of Terraform Enterprise, your organization must consider the deployment of a Redis instance in your private datacenter as this is a hard requirement for Active/Active. Customers for whom this proves impossible are recommended to consider a public cloud deployment, or HCP Terraform if SaaS is a viable option for your organization.

To avoid arranging product ownership and ongoing maintenance of Redis, the only self-managed alternative is to use the Single Instance + External Services deployment topology, but this means there being only one Terraform Enterprise container in the instance. In certain circumstances, if the container can be restarted by an orchestrator such as Kubernetes fast enough for your needs, and/or your RTO dictates this as a viable possibility, then Redis can be avoided, however, in our experience, your business will require higher availability and lower RTO over time, so at least, forward planning for Redis support on-premise is recommended.

PostgreSQL

Many customers use alternatives to PostgreSQL as their strategic database solutions. Terraform Enterprise depends specifically on PostgreSQL, which means a private cloud deployment requires that your organization has a pattern for the deployment and production management of PostgreSQL at the versions listed on our website.

Also be aware that there are specific schema requirements in the above link, so you are advised to liaise in the first instance with your DBA team so that all parties are aware of the requirements at the outset.

S3-compatibility

In addition to Redis and PostgreSQL, Terraform Enterprise requires S3-compatible storage to operate. In a private datacenter scenario, this means adopting a third-party technology to present an S3-compatible API to Terraform Enterprise.

Traditionally, HashiCorp has pointed customers who do not already have a solution in place to the most-used examples, however it is not for us to comment on the suitability of one solution for your business over another.

From working with our customers, we see significant success with private cloud deployments of Terraform Enterprise using either Dell ECS or MinIO as the object storage tier. Our MinIO setup guide for Terraform Enterprise is detailed here. If your organization has a pattern for on-premise S3-compatible object storage, our recommendation is to use that.

Compatibility and components sizing

Compute

We recommend deploying Terraform Enterprise on one the following operating systems:

• Red Hat Enterprise Linux 7.9 or 8.7 (also see RedHat Linux requirements). Our current deployment module currently deploys RHEL 7 but this can be overridden in case of organizational requirement.
• Ubuntu 22.04

Sizing

Each instance of the Terraform Enterprise application service runs as a container in Docker/Kubernetes. The following compute requirements are appropriate for most initial production deployments, or for development/testing environments.

• CPU - 4 vCPU
• Memory - 16 GB RAM
• Disk - 1TB

We recommend configuring the CPU and memory availability identically for each Terraform Enterprise application instance. Ultimately, actual computational workload requirements reflect your organization's needs, but these are safe limits HashiCorp uses in scaling tests.

Many of our scaled customers use 8 vCPU and 32GB RAM as an initial production specification. Either way, expect to scale past this as you begin to scale out.

When allocating CPU and memory resources, it is essential to consider the computational needs of the VM(s) and any adjacent containers running in the same orchestration instance. Thus, the CPU and memory allocation of your VM instances must be higher than the maximum configured for the Terraform Enterprise application. Consult the Terraform Operations Guides for more information on ongoing right-sizing and scale out of your Terraform Enterprise instances.

The disk space used by the object store will depend on a few things:

• The number of workspace runs - each run generates objects in the object store irrespective of whether the run succeeds or fails.
• The size of configuration directories submitted to the API during Terraform runs. Terraform runs aggregate the configuration directory and sends it to the API which, in turn, commit it to encrypted object storage using an internal process called archivist. We recommend limiting your configuration repositories to containing only necessary files needed for the Terraform workspace run to proceed. A 50MB .tgz file in a configuration directory will needlessly add 50MB to your object store and wastes hosting costs.

Disk

The performance of disk storage available to the Linux operating environment plays a role in the performance of Terraform Enterprise. We recommend using a disk with a minimum of 3000 IOPS to support data operations.

Terraform Enterprise requires a disk with at least 40GB of available space; for Docker deployments this space should be available to /var/lib/docker. This limit does not apply to Kubernetes deployments.

PostgreSQL

Sizing

We recommend the following minimum specification for the DB server.

• CPU - 4 core
• Memory - 32 GB RAM
• Disk - 2TB

Redis cache

The supported Redis cache versions are:

• Release 6.x
• Release 7.x

Redis Cluster and Redis Sentinel are not supported.

Sizing

We recommend the following characteristics for the cache.

• CPU - 4 core
• Memory - 16 GB RAM
• Disk - 500GB

Monitoring PostgreSQL and Redis servers

As a best practice we recommend that CPU, memory, available disk space and disk IO are closely monitored using your standard organizational machine telemetry solution. Create alerts for threshold breaches of 50% and 70% utilization. If utilization of any of the parameters exceeds 70% consistently, consider increasing the resource to ensure optimal performance. We recommend ongoing telemetry reporting, so the team owning the instance are aware of how quickly resources are being consumed.

Networking requirements

Terraform Enterprise has a number of ingress and egress requirements.

Please refer to the network requirements page for the latest information.

If a corporate proxy service is filtering outbound Internet traffic, then the destinations detailed in the above link need to be added to your allow-list.

Terraform Enterprise works with both layers 4 and 7 load balancers, but consult the architecture section in this documentation for recommendations on TLS management which apply equally here.

Installation process - Docker

Write an automated Terraform Enterprise installation process which conforms to your organizational requirements and is based on the following. Bash has been used for illustration; the tasks required should be written into automation so that each machine starts up and installs the software the same way.

We recommend reading through the whole document prior to running the steps manually as part of your planning for automation.

Prepare requirements

The following list summarizes the infrastructure resources required prior to the deployment of Terraform Enterprise.

1. VMs for installation of Terraform Enterprise
2. PostgreSQL database
3. Redis cache
4. S3-compatible object storage
5. Terraform Enterprise license file (if you already have a license file but are currently on a Replicated-backed instance, you will need a replacement file - the contract will be the same)
6. TLS certificate bundle
7. Load balancer

Tasks

As root, perform the equivalent of these commands:

sudo mkdir -p /opt/tfe/certs
cd !$

Put the PEM-encoded server certificate, private key and CA bundle files in this directory. For the sake of this installation, we assume the files are called cert.pem, key.pem and bundle.pem.

tfe
`-- certs
    |-- bundle.pem
    |-- cert.pem
    `-- key.pem

::: warning

Terraform Enterprise cannot use a private key that is protected by a passphrase.

:::

Create a Docker Compose configuration

Write a Docker configuration with the consolidated information about your Active/Active environment. Base this compose.ymlfile on this one. Note that you will be using on-premise S3-compatible object storage, so use the TFE_OBJECT_STORAGE_* variables listed in the link as you would were you to deploy Terraform Enterprise on AWS.

::: warning

The compose.yml contains sensitive information that needs to be protected. Please treat this file as you would any other company secret.

:::

In the configuration properties, we want to highlight the following:

• Treat the TFE_ENCRYPTION_PASSWORD value as a sensitive secret. While it can be defined arbitrarily, we recommend using a password generation utility or process to follow corporate policies. Terraform uses this password to decrypt the Vault token stored in the database; the Vault token is used to do all subsequent encryption/decryption processing - see below.
• Terraform Enterprise uses an embedded instance of HashiCorp Vault for data security operations. When identifying the TFE_VAULT_CLUSTER_ADDRESS for the Vault instance, point to the private IP of the VM. The internal instrumentation of Vault performs a discovery to find peer instances and add to a private cluster so the IP addresses of all nodes in the cluster need to be accessible to all the other nodes.
• The TFE_DATABASE_HOST points to your PostgreSQL database cluster endpoint name.
• The TFE_REDIS_HOST points to the endpoint of the primary node in the Redis Replication Group.
• The TFE_LICENSE entry requires a raw string from your Terraform Enterprise license file. The license is distributed as a JSON formatted file. The license string is the data.attributes.signed_string object.

This is not an exhaustive list of configuration options. Please refer to the configuration reference for documentation about all the configuration options.

Write the compose.yml to a location where your nascent application machines can access it - each of them will need to use it for the installation.

Configure systemd

We recommend using the following systemd manifest to run docker compose on system start, so that if the machine reboots, Terraform Enterprise will automatically start. Create /etc/systemd/system/terraform-enterprise.service containing

[Unit]
Description=Terraform Enterprise Service
Requires=docker.service
After=docker.service network.target
 
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/etc/terraform-enterprise
ExecStart=/usr/local/bin/docker compose up -d
ExecStop=/usr/local/bin/docker compose down
TimeoutStartSec=0
 
[Install]
WantedBy=multi-user.target

then run

systemctl enable --now terraform-enterprise

Download the Terraform Enterprise image

First, login to the Terraform Enterprise container image registry using

cat <PATH_TO_HASHICORP_LICENSE_FILE> \
| docker login --username terraform images.releases.hashicorp.com --password-stdin

If you do not receive the output Login Succeeded using the provided license, please contact your HashiCorp representative for support.

Next, pull the container image. Use the form docker pull images.releases.hashicorp.com/hashicorp/terraform-enterprise:<vYYYYMM-#> where <vYYYYMM-#> is the release required for example:

docker pull images.releases.hashicorp.com/hashicorp/terraform-enterprise:v202309-1

Check the releases page for the suitable version. We recommend using last month's latest release - if it is October, get v202309-x where x is the highest listed integer.

If necessary, you can pull the image and push it to your private Docker registry. When placing the image in a private Docker registry ensure to re-tag the image to support your hierarchical organization standards.

Run docker compose

Place the compose.yml in your working directory. When done, the directory structure must look like this:

tfe
|-- certs
|   |-- bundle.pem
|   |-- cert.pem
|   `-- key.pem
`-- compose.yml

Each node will then need to run

sudo docker compose up --detach

You should then see output messages indicating the progress of the running process. Logs can be followed using this in a separate shell:

docker compose logs --follow

Run a health check

Terraform Enterprise provides a /_health_check endpoint on the instance. If Terraform Enterprise is up, the health check returns a 200 OK.

The /_health_check endpoint operates in 2 modes:

• Full check with /_health_check?full=1
• Minimal check with /_health_check

With a full check, the service will attempt to verify the status of internal components and PostgreSQL, in contrast to a minimal check which returns 200 OK automatically after a successful full check.

To run a health check, use:

sudo docker exec terraform-enterprise curl http://127.0.0.1:9292/_health_check?full=1

The command returns a JSON encoded message as follows:

{"postgres":"UP","redis":"UP","vault":"UP"}

Use the following or equivalent in your code in order to pause the deployment process until all Terraform Enterprise processes are healthy:

while ! docker compose exec terraform-enterprise curl http://127.0.0.1:9292/_health_check?full=1;
do
  echo "Waiting for Terraform Enterprise to be ready"
  sleep 30
done

The default behavior of this endpoint is to perform a full check during startup of the instance, and minimal checks after Terraform Enterprise is active and running. The Terraform Enterprise application supports a practical tfe-health-check-status command to provide full visibility. You can run the command in the VM as follows:

sudo docker exec terraform-enterprise tfe-health-check-status

When successfully started, Terraform Enterprise returns messages as follows:

checking: Archivist Health Check...
|  checks that Archivist is up and healthy
|- ✓ PASS
 
checking: Terraform Enterprise Health Check...
|  checks that Terraform Enterprise is up and can communicate with Redis and Postgres
|- ✓ PASS
 
checking: Terraform Enterprise Vault Health Check...
|  checks that Terraform Enterprise can connect to Vault and is able to encrypt and decrypt tokens
|- ✓ PASS
 
checking: Fluent Bit Health Check...
|  checks that the configure Fluent Bit server is healthy
|- SKIPPED
 
checking: RabbitMQ Health Check...
|  checks that RabbitMQ can be connected to and that we can send and consume messages
|- SKIPPED
 
checking: Vault Server Health Check...
|  checks that the configured Vault Server is healthy
|- ✓ PASS
 
  All checks passed.

When forcing a full check, take extra caution as every call makes requests to all external services, increasing system load and latency. This is not a concern during the initial installation stages as there are no in-flight operations expected.

Retrieve your initial admin creation token

When you have deployed all Terraform Enterprise application instances, you can retrieve the initial admin creation token. Choose one of your EC2 Instances and run the following command in a terminal session:

docker compose exec terraform-enterprise retrieve-iact

The command returns your initial admin creation token. The token is sensitive and must be protected as part of the handover process to the operating teams. The form is:

5db5c3ae5be1599804bc614ec976e1a64ad1cadf76fec518d61ed9d04ef565eb

To view the startup web page, open a web browser and navigate to the Terraform Enterprise URL.

The syntax for the URL endpoint is: $URL/admin/account/new?token=$TOKEN

For example:

https://alb_entry_point.your_domain.com/admin/account/new?
token=5db5c3ae5be1599804bc614ec976e1a64ad1cadf76fec518d61ed9d04ef565eb

When the URL is loaded, a form is presented to the operator. The first user created in Terraform Enterprise becomes the site administrator. The only reason to complete this form is when the deployment team is responsible for post-installation tasks of the Terraform Enterprise deployment team.