Terraform workflows

By adopting IaC practices, organizations have gained agility, efficiency, and reliability in deploying cloud-native resources. This section details integration of version control systems (VCS) and building efficient pipelines for Terraform.

Streamlining IaC

As you automate Infrastructure as Code (IaC) provisioning and management, you can significantly streamline the process, similar to automating source code build and deployment with Continuous Integration/Continuous Deployment (CI/CD) pipelines. To simplify this automation and integrate with Version Control Systems (VCS) repositories, HCP Terraform and Terraform Enterprise offer powerful features.

You can take advantage of the VCS integration feature in HCP Terraform and Terraform Enterprise, which uses webhooks to directly integrates with VCS repositories. This allows you to seamlessly implement a gitops workflow without the need for custom pipelines. We highly recommend using this method as it offers a straightforward and efficient approach.

In situations where you prefer to utilize CI/CD pipelines, HCP Terraform and Terraform Enterprise provide pipeline templates to facilitate the integration. These templates are designed to make it easier to incorporate IaC code from VCS repositories into HCP Terraform and Terraform Enterprise. Consider using CI/CD pipelines when the VCS integration might not entirely meet your specific requirements. We currently provide templates for GitHub and GitLab, but you can use the underlying workflow tooling to build an integration for other CI/CD solutions.

Overview

HCP Terraform and Terraform Enterprise (TFE) play a vital role as CI/CD platforms for Infrastructure as Code (IaC), empowering organizations to manage their infrastructure efficiently. They support CI tools by running plans and policy checks to ensure changes meet functional and compliance requirements. They further handle CD by applying changes and conducting continuous validation and drift detection, making them ideal CI/CD tools for infrastructure management.

Embracing Infrastructure as Code allows you to define your infrastructure using source code, akin to traditional software systems. Storing your configurations in version control grants you the ability to conduct audits, testing, and continuous delivery, addressing the complexities of deploying cloud-native resources effectively.

In a constantly evolving infrastructure landscape, responsiveness and reliability are key. When selecting a workflow, consider deployment frequency, code change frequency, and the level of automation required, while adhering to established guardrails.

While our primary focus here is on designing a pipeline workflow using the VCS-driven approach with HCP Terraform and Terraform Enterprise, we will also briefly touch on CLI-driven and API-driven workflows to provide a comprehensive understanding of Terraform's capabilities. However, the VCS-driven workflow is the recommended path for most organizations due to its seamless integration and collaborative benefits.

Personas

When constructing the workflow, it is advisable to involve only a minimal number of contributors in the pipeline development process. Depending on the size and scale of your organization, certain roles may be combined to take on additional responsibilities.

VCS-driven workflow

When it comes to managing your infrastructure as code, Terraform offers a VCS-driven workflow that streamlines the process and enhances collaboration among your team members. This approach automatically triggers runs based on changes made in your VCS repositories, making it easier to keep track of modifications and ensuring that your infrastructure configuration stays up-to-date.

Embracing Terraform's VCS-driven workflow brings with it a host of valuable benefits. By establishing shared repositories as the definitive source of truth, your team can work concurrently on the same infrastructure code, promoting seamless collaboration. Additionally, this workflow eliminates the need for additional Continuous Integration (CI) tooling, simplifying the process and making it more accessible to all team members. Moreover, you can effortlessly associate this workflow with webhooks on your preferred VCS providers, further streamlining the process of triggering runs and ensuring efficient synchronization between code changes and infrastructure updates.

Prerequisite(s)

Before proceeding with the VCS-driven workflow, ensure you complete the following pre-work for each category:

1. VCS repository:

   • Set up a Version Control System (VCS) repository that contains the source code for the deployment.

2. VCS authentication:

   • Establish VCS authentication to enable Terraform's access to the repository securely.

3. VCS permissions:

   • Define the required VCS permissions for individuals, specifying appropriate access levels such as read-only, merge, etc.

4. Workspace:

   • Define workspace naming conventions to maintain consistency.
   • Define workspace permissions, ensuring that the right users have the necessary access for collaboration.
   • Configure workspace settings, including enabling speculative plans, auto-apply, and other relevant options.
   • Decide on a specific git tag format for auto-apply to streamline the deployment process.

Workflow overview

These are the high level steps one would take to implement a successful VCS-driven workflow in Terraform.

1. Configure VCS Integration: Establish a VCS integration for your organization to connect HCP Terraform and Terraform Enterprise with your Version Control System.
2. Connect Workspace to Repository Branch: Link your workspace to the desired branch in your VCS repository, enabling automatic synchronization of code changes as they are pushed.
3. Adapt Branching Strategy: Adopt a suitable branching strategy, such as git flow, to effectively manage different stages of development and deployment within your VCS repository.
4. Enable Speculative Plan Runs: Automate speculative plan runs for each branch, allowing you to preview changes and validate configurations without impacting the live infrastructure.
5. Define PR Process: Establish a Pull Request (PR) process (which essentially means that branch protection is enabled and any changes will need to come in via a PR) with the necessary approvers to ensure thorough code review and approval before deploying changes.
6. Configure Automatic Run Triggers: Set up automatic run triggers for the plan phase based on git tag pushes, streamlining the deployment process and enhancing efficiency.
7. Deploy with 'Action': Utilize an 'Action' configuration to facilitate deployment based on the git tag push. You can configure this 'Action' for either plan or apply mode, depending on your specific requirements.

These essential steps for setting up a VCS-driven workflow will be discussed in more depth in the subsequent sections. We'll explore each step thoroughly, providing detailed insights and practical guidance to help you effectively implement and streamline the workflow within your organization.

VCS integration

Incorporating the Version Control System (VCS) provider is crucial to establishing an automated workflow that predicts, plans, and initiates runs seamlessly. In the VCS-driven approach, each workspace is linked to a specific VCS repository, ensuring a clear association between branches and workspaces. To access configurations from the VCS, Terraform requires permissions for the following tasks:

• Access a list of repositories: This enables you to search for repositories when creating new workspaces. Register webhooks with your VCS provider: Webhooks keep HCP Terraform informed about new commits to your chosen branch.
• Download repository contents at a specific commit: This action allows Terraform to execute with the specified code. HCP Terraform uses webhooks to monitor new commits, pull requests and tags. When a new commit is pushed to the repository branch, it triggers an automatic run based on the configured trigger. Similarly, initiating a pull request or merge request to the branch will activate speculative plans.

VCS connection

Integrating the VCS provider is essential for the workflow to automatically predict, plan, and initiate runs. HCP Terraform uses the OAuth protocol to authenticate with VCS providers, and each provider may have distinct methods and steps to configure this connection. For comprehensive details on setting up the connection with your specific VCS provider, please refer to the respective guide.

Connecting your Terraform workspace to the desired branch in your VCS repository is a key step in the VCS-driven workflow. By linking the workspace and the branch, you enable automatic synchronization of code changes as they are pushed to the repository. This means that every time a new commit is made to the associated branch, HCP Terraform will automatically trigger a run based on the configured trigger.

HCP Terraform provides three main options to specify which changes trigger runs in your repository:

1. When any changes are pushed to a specific branch

2. When changes are made to particular files within a specified branch (using the trigger patterns)

3. Any tags to a repo matching a pattern (which is what is described in option 1). Branches are irrelevant for this option. A single change in a repository can cause changes in multiple workspaces if customers are mapping a given repo-branch combination to multiple workspaces: This option instructs Terraform to begin new runs only for changes that have a specific tag format. You can choose between various tag format options:

   • Semantic Versioning: It matches tags in the popular semantic versioning format. For example, 0.4.2.
   • Version contains a prefix: It matches tags which have an additional prefix before the semantic versioning format. For example, version-0.4.2.
   • Version contains a suffix: It matches tags which have an additional suffix after the semantic versioning format. For example, 0.4.2-alpha.
   • Custom Regular Expression: You can define your own regex for HCP Terraform to match against tags.

Auto apply

Terraform offers an "auto apply" option, which will apply changes from successful plans without prompting for approval (this is irrespective of way the plan is triggered - VCS, API etc). This feature is especially useful in non-interactive environments like continuous deployment workflows, where you want to ensure that no one can change your infrastructure outside of your automated build pipeline. Enabling auto apply reduces the risk of configuration drift and unexpected changes to the infrastructure.

Branching strategy

Now that we've discussed how to configure the VCS integration and establish the connection between HCP Terraform and your Version Control System, let's explore how to effectively manage your workspaces by utilizing a defined branching strategy. Having a well-defined branching strategy is important as it allows for effective management of different stages of development and deployment, providing clarity and control over the code changes introduced at each phase. As an overview, here are workspace configuration guidelines that will assist you in maintaining a well-organized, scalable, and efficient Terraform workflow:

• Unique Repository+Branch Mapping: The recommended pattern is to assign each workspace to a unique combination of a repository and branch. Whenever changes are made to the associated repository and branch, a Terraform run will automatically trigger for that specific workspace.
• Subfolders within Repository: While subfolders can exist within the repository, it's important to ensure that all resources in the repository are managed by the same workspace. This practice promotes isolation and organization, making it easier to manage and maintain Terraform configurations.
• Branches for Related Workspaces: Utilize different branches to represent related workspaces. For example, you can have a branch named main for production and another branch named dev for development. This approach helps to separate and manage configurations for different environments efficiently.
• Pull Request (PR) Workflow: When a PR is opened towards a specific workspace branch, the PR must only contain information about that particular workspace. This focused approach allows for clearer review and approval of changes specific to the intended workspace.
• Release Workflow: The workflow for releasing changes follows standard development practices. Developers make changes on a feature branch, which is later merged into a long-lived branch like dev. Finally, the changes from the dev branch are merged into the main workspace, enabling a controlled and streamlined release process.

Recommendation

Consider adopting a branching strategy similar to GitHub Flow, especially if GitHub is your primary version control tool. This model is based on trunking but utilizes short-lived branches instead of feature tags. With this approach, you create a branch from the main branch, make your changes (in our Terraform use case, these changes would be within the environment's specified workspace folder), commit the changes, push a pull request, merge the changes into the main branch, and then delete the branch. The main objective of this model is to have short-lived branches, ensuring that the main branch always reflects the latest features or changes at any given time.

As a general approach, we suggest aligning the dev, stage, and production branches with their respective workspaces for applications. This alignment promotes a clear and structured workflow, enabling smooth transitions between different development stages and deployments.

Here are the steps that align with the GitHub Flow Git Branching Model:

1. Start by creating a repository with three branches - dev, stage, and prod. Commit your initial code in the dev branch.
2. For each feature development, create a separate feature branch off the dev branch. For example: feature/app1-web-tier.
3. Make your changes and commit them to your branch. Ensure that each commit contains an isolated and complete change for easy reversion and feedback.
4. Push your changes to your feature branch and create a pull request.
5. Address any comments, issues, tasks, or questions from reviewers during the pull request review process.
6. The reviewer will merge your pull request, incorporating your feature branch into the dev branch.
7. The final reviewer will perform a squash commits and merge, streamlining the changes into the dev branch.
8. The feature branch can be deleted after merging to maintain only long-lived branches for each environment (dev, stage, and prod).

Speculative plan

Now that we have looked at the recommended branching strategy, let's explore the code update workflow that triggers plans or runs. In this process, speculative plans play a crucial role as they allow you to preview and validate changes before merging them into the main branches. This ensures that your infrastructure remains stable and reliable. Here are the steps for the code update workflow, taking advantage of the recommended branching strategy and utilizing speculative plans:

1. Create a feature branch from the dev branch to work on specific changes.
2. Test and validate the features in the feature branch.
3. Open the PR. This will trigger a speculative run.
4. Review, approve and merge the changes into the dev branch.
5. Repeat the steps 3 and 4 for stage branch and then the main production branch.
6. Delete the feature branch to avoid the maintenance of long-lived feature branches.

Speculative plans are a valuable feature for reviewing proposed changes. Terraform can automatically run speculative plans for pull requests or merge requests. You can view speculative plans in a workspace's list of normal runs. Additionally, if you are using GitHub,Terraform adds a link to the run in the pull request itself, along with an indicator of the run's status. When you update a pull request, Terraform performs new speculative plans and updates the link.

Although any contributor to the repository can see the status indicators for pull request plans, only members of your Terraform organization with permission to read runs for the affected workspaces can click through and view the complete plan output.

Whenever a pull request is created or updated, Terraform checks whether it should run speculative plans in workspaces connected to that repository, based on the following rules:

• Only pull requests that originate from within the same repository can trigger speculative plans.
• Pull requests can only trigger runs in workspaces where automatic speculative plans are allowed. You can disable automatic speculative plans in a workspace's VCS settings.
• A pull request will only trigger speculative plans in workspaces that are connected to that pull request's destination branch. In order to trigger an apply run, the PR must be closed and merged into the branch, which means statuses are no longer being registered.

HCP Terraform does not update the status checks on a pull request with the status of an associated apply. This means that a commit with a successful plan but an errored apply will still show the passing commit status from the plan.

Additional workflow options

While the recommended VCS-driven workflow offers seamless collaboration and automation through integration with your Version Control System, it's good to consider alternative approaches that might better suit your organization's specific needs. The VCS-driven workflow provides unparalleled flexibility and enables teams to leverage their existing CI/CD pipelines effectively. However, in certain scenarios, you might find it beneficial to explore other options such as the API-driven and CLI-driven workflows. These alternative approaches have their advantages and can be valuable additions to your Terraform workflow.

API-driven workflow

When working with HCP Terraform and Terraform Enterprise, the API-driven workflow offers complete control over the IaC process, making it highly compatible with custom CI/CD pipelines utilizing the HCP Terraform and Terraform Enterprise API. If your requirement involves seamless integration with external CI tools, an API-driven workflow can be the most suitable approach, providing flexibility to utilize your current CI tools for tracking state and maintaining continuous linkage between Terraform and your code. As the most advanced approach, this workflow is often adopted by large organizations with complex requirements, either directly or as an evolution from the CLI-driven workflow.

The API-driven workflow offers distinct advantages over the recommended VCS approach, including unparalleled flexibility for integration with existing CI/CD pipelines, additional testing capabilities, and full integration with third-party platforms. It also enables centralized code management, retains current CI/CD configurations, and leverages existing programming languages for a more tailored Terraform workflow.

API workflow overview

The high-level API-driven workflow utilizing HCP Terraform and Terraform Enterprise API endpoints includes the following steps:

1. Define variables for organization, workspace, and projects using API tokens, usually team tokens.
2. Prepare the configuration file for upload using the configuration version API.
3. Solicit the workspace ID using the configuration version API.
4. Create a configuration version using the configuration version API.
5. Upload the configuration version using the run apply API.

Pipeline templates

The API-driven workflow can be further enhanced by leveraging pipeline templates, which bring numerous benefits to integrating HCP Terraform and Terraform Enterprise with various CI/CD platforms. These templates provide clear and predefined examples, making the integration process straightforward and less time-consuming. With a quick-start approach, organizations can streamline their integration process, accelerating the deployment of Terraform in their CI/CD workflows while ensuring consistency across the pipelines. By utilizing existing templates, organizations save effort and resources that would otherwise be spent on developing custom API-driven workflows. Moreover, these prescriptive templates are designed to cater to various CI/CD platforms, ensuring seamless compatibility and smooth integration with existing tools. As an added advantage, the templates are continuously maintained and updated by the Terraform community, allowing organizations to easily benefit from the latest improvements and best practices, keeping their Terraform workflow up-to-date and efficient.

When integrating Terraform into your existing CI/CD pipelines, consider the convenience and efficiency of leveraging prescriptive pipeline templates. These templates provide explicit examples of how to integrate with HCP Terraform and Terraform Enterprise on various CI/CD platforms, offering a quick and straightforward way to get started without the need to develop templates from scratch. By following these steps, you can smoothly incorporate Terraform into your CI/CD workflows:

1. Choose a pipeline template that aligns with your CI/CD platform and meets your specific integration needs with HCP Terraform and Terraform Enterprise.
2. Tailor the template to suit your organization's requirements by adjusting configurations, adding or removing steps, and integrating it with other tools in your CI/CD pipeline.
3. Integrate the customized template into your CI/CD pipeline and thoroughly test its functionality to ensure it aligns with your desired workflows.

Examples

HashiCorp offers GitHub Actions that integrate with HCP Terraform and Terraform Enterprise APIs, empowering you to create customized CI/CD workflows tailored to your organization's specific needs.

A typical GitHub Action workflow includes generating a plan for each commit to a pull request branch, which can be reviewed in HCP Terraform or Terraform Enterprise. Once you update the VCS branch, the configuration is automatically applied.

Upon configuring the GitHub Action, you can create and merge a pull request to test the workflow. HCP Terraform's built-in support for webhooks facilitates this generic workflow.

Moreover, by leveraging HashiCorp's Terraform GitHub Actions, you can design a custom workflow with additional steps before or after your Terraform operations, enhancing your automation capabilities.

For additional information and resources, you can refer to the following documentation:

• Announcement
• Tutorial

CLI-driven workflow

With Terraform's CLI-driven workflow, you can transition from using OSS CLI Commands to HCP Terraform Agents for triggering the build process. This workflow offers the flexibility to iterate quickly on your configuration and work locally, allowing for efficient development and testing. Moreover, it simplifies integration with CI/CD tools, enabling the addition of testing steps and enhancing deployment reliability, streamlining the process compared to the VCS-driven approach. Additionally, the CLI workflow promotes centralized code management, fostering effective collaboration and version control, which is especially valuable for organizations with complex requirements. Furthermore, it permits convenient local development of workspaces, minimizing the need for frequent code commits and enhancing the overall development experience. For organizations with existing CI/CD configurations, transitioning to the CLI workflow can be straightforward, allowing them to retain their current setups and reducing the need for major adjustments. Lastly, the CLI workflow provides faster feedback to practitioners, enabling greater flexibility in deploying changes across different environments, making it an attractive alternative to the VCS-driven approach for those who require rapid iterations and deployments.

Workflow overview

This high-level CLI-driven workflow includes the following steps:

1. Generate a CLI token for use as a CI/CD tool or local user.
2. Create a workspace and update the backend configuration block.
3. Construct CI/CD pipelines to include pre-plan checks like linting, security scanning, and unit testing.
4. Trigger the plan using the terraform plan, apply, or destroy CLI command, whether invoked by a human user or a CI/CD script.
5. Configure your CI/CD tool to automatically trigger plans based on branch or directory changes.
6. Set up different pipelines for pull requests, pushes to specific branches, etc. After making and validating code changes, merge pull requests to the main branch.