Run Tasks

HCP Terraform run tasks let you directly integrate third-party tools and services at certain stages in the HCP Terraform run lifecycle. You can use run tasks to validate Terraform configuration files, analyze execution plans before applying them, scan for security vulnerabilities or compliance requirements, perform detailed cost analysis, or perform other custom actions.

Run tasks send data about a run to an external service at specific run stages. The external service processes the data as appropriate, can evaluate whether the run passes or fails, and sends a response back to HCP Terraform. HCP Terraform uses this response and the run task enforcement level to determine if a run can proceed. The external service can store and monitor the run-related data for continuous review and monitoring and for historical purposes.

The run tasks functionality sends an API payload to the external service endpoint. This payload contains the information Run tasks send an API payload to an external service. This payload contains the pertinent information related to the workspace and run, and includes a callback URL which the external service uses to return a pass or fail status and messaging to HCP Terraform.

Benefits of Run Tasks

Your teams can integrate third-party tools into the HCP Terraform workflow during any of the run stages: pre-plan, post-plan/pre-apply, and post-apply. By allowing for this real-time integration during the run workflow, conditions can be set for runs to pass and continue, or fail and terminate, with only a brief interruption in the run flow.

Run tasks can take the place of some Sentinel policies that you would otherwise have to develop and manage. The replacement would be an integration with a third party product well-suited to testing the specific intricacies of the infrastructure provisioning concerns they support as experts. This can reduce development and testing time while providing more stringent and accurate security and compliance testing.

Run tasks now support a rapidly growing number of integrations with many well-known tools that you may already be taking advantage of. And if you find that an internal or external product or requirement is not currently supported in the way you require, you can develop a custom run-task to meet the need, or possibly request the product owner to explore that as a valuable add-on feature to their offering. Some examples of the specific use cases for post-plan run task integrations available and currently in use are:

• Security and regulatory compliance: Ensure that you’re not provisioning misconfigurations that cause security or compliance issues using tools like Palo Alto Networks Prisma Cloud, Zscaler, Snyk, Tenable,, Sophos, Aqua Security, Firefly, and
• Cost control: Provide visibility into the costs of infrastructure prior to any change using Infracost, Vantage, or Kion
• Visibility: Enhanced resource visualization with Pluralith
• Image compliance: Verify that only approved and tracked golden images are used with the HCP Packer run task

For a complete list of available run tasks please refer to public Terraform Registry.

Overall flow for implementing Run Tasks

The overall flow of setting up to execute a run task against a workspace, or set of workspaces is as follows:

• Select the desired run task from the registry, review the requirements and documentation, and ensure connectivity can be made in a two-way fashion with the HCP Terraform Platform.
• Connectivity between HCP Terraform platform and the run task endpoint must be established and verified. This will likely require network and security modifications to open the path.
• The desired run task must be “created” in the Terraform Organization in order to be available for assigning to workspaces.
• This is an action of configuring the run task and connecting the endpoints on both sides and testing the communication path has been established.
• The run task must be associated with a workspace and configured as to what stage to execute in and whether the outcome will be handled as Advisory or Mandatory.
• The run task will execute as part of the normal run cycles for that workspace and the results can be examined in the same way run completions are validated.

Overall flow for Run Tasks within a run

Run tasks are invoked before and after the plan and apply stages of a Terraform run. The information that is available at each stage is presented to the run task when invoked. At the pre-plan stage, the information available will mainly be the code and other attributes; at post-plan and later stages, more information associated with the results of the plan will be available.

Selecting the stage to associate a run task to will vary for a number of factors. The pre-plan stage would be used when the run task is going to examine the code and make a determination if entering the plan stage should even be allowed. The other stages primarily depend on how far you want a plan to go before being stopped by a failure result of the run task and what states of provisioning you are able to accept.

For example, a post-plan or pre-apply run task will examine the plan and determine whether an apply should be allowed. A post-apply would be used mainly to test and possibly gather/store information related to the infrastructure that has been provisioned. Check the documentation for the run task being configured if there are particular stages required or suggested to satisfy its requirements.

Multiple run tasks can be defined at a single stage. They will execute in sequence and in no particular order that could be depended on (do not assume any dependencies between run tasks). Once all run tasks complete for a stage, the continuation of the run is determined by the most restrictive enforcement level in each associated run task.

For example, if a mandatory task fails and an advisory task succeeds, the run fails. If an advisory task fails, but a mandatory task succeeds, the run succeeds and proceeds to the next stage. Regardless of the exit status of the run tasks, HCP Terraform displays the status and any related messages are recorded and appear in the run results.

Available Run Tasks

The common implementation of run tasks will be to implement an existing run task that integrates with a specific complementary product, usually provided and supported by a HashiCorp technology partner that provides that product. These are published in the public Terraform Registry where you can then view the most up-to-date list of available run tasks with links to the partner’s usage documentation. Other vendors or community members may offer run tasks for specific integrations, but they will not be published to the public Terraform Registry, and should be handled as any other open source software.

In addition to using existing technology partner integrations, HashiCorp Terraform customers can build their own custom run task integrations. The process of developing custom run tasks is discussed in a separate topic in this documentation set.

Run Task specific requirements

Each run task provides specific configuration requirements for integration with HCP Terraform. The documentation supplied by the run task's owner should include all necessary details for setup. You should carefully review these requirements and prepare accordingly to complete the configuration and ensure proper connectivity. This process typically involves adjustments to network and security settings and generating tokens to establish secure connections between the services.

Terraform version

You can only assign run tasks to workspaces that use a Terraform Core version of 1.1.9 and later. You can later downgrade a workspace with existing runs to use a prior Terraform version without receiving an error related to the run task association. However, HCP Terraform will no longer trigger associated run tasks during the run workflow.

Permissions required for managing Run Tasks

To create a run task in an Organization, you must have a user account with the Manage Run Tasks permission. To associate run tasks with a workspace, you need the Manage Workspace Run Tasks permission on that particular workspace.

Manage Run Tasks permission

Manage Workspace Run Tasks permission

Managing Run Tasks in an organization

Run tasks can be configured and managed in any of the standard ways you can configure HCP Terraform in general. The methods for configuration are:

• HCP Terraform UI
• HCP Terraform APIs
• TFE Provider

We strongly encourage you to use a repeatable automation method over using the manual method of managing them through the HCP Terraform UI. The UI can be useful for initial experimentation and testing, however, a reusable automation will provide an easy, repeatable, and auditable way to manage the configurations. The TFE Provider provides the cleanest DevOps implementation, by not only being repeatable and portable, but will also track the state and history and only apply new changes on each invocation.

The workflow for creating and maintaining run tasks and managing the association of run tasks to workspaces is the same regardless of the configuration method used. The documentation for creating run tasks and associating to workspaces using the UI, describe the flow and the details of the actions and their attributes that must be supplied and can be used for navigating the other methods:

• Creating a Run Task
• Associating a Run Tasks with a Workspace

Please note that removing a run task from a workspace does not delete it from the organization. To fully delete the run task from the organization, it must be detached from all workspaces before it can be removed. Additionally, it's important to remember that a run task cannot be deleted from the organization while it is still linked to any workspace.

Using HCP Terraform UI

These tutorials provide an example of setting up run tasks in general, and use the HCP Terraform UI method:

• Configure Snyk run task in HCP Terraform
• Set up HCP Terraform run task for HCP Packer

Using Run Tasks API

Run tasks can be managed with automation using the Run Task API, built specifically for this purpose. HashiCorp recommends automating the management of these tasks, preferably by including the automatic association of run tasks in your onboarding and Landing Zone type automations that are already managing the creation and other attributes of workspaces.

Since some run tasks will be applicable to sets of workspaces and possibly all workspaces in an organization, associating them manually will be time-consuming. For consistency and security, we recommend automating run tasks whenever possible.

The complete documentation for the Run Task API is located here Run Tasks - API Docs - HCP Terraform.

The API includes all of the standard CRUD requests that allow you to manage the run tasks and the associations of run tasks to workspaces. You can develop the processes that invoke the APIs using the standard tools or languages that you use for such tasks.

Managing run tasks with APIs requires API tokens generated from the same permissions as managing them through the UI. Information on working with HCP Terraform tokens is located here: API Tokens - HCP Terraform. .

The Run Task API has two distinct sets of endpoints: one for managing the run tasks within an organization and the other for managing the association of run tasks to workspaces.

The API endpoints for creating and managing run tasks in an organization are straightforward. The initial creation and list endpoints start are based at the organization level.

The remaining endpoints are based at the task level as they rely on the run task ID that was assigned as part of the initial creation.

The API endpoints for associating and managing run tasks for a workspace are also straightforward. All of the endpoints start from the workspaces endpoint, as the run task associations are by workspace level and require a workspace ID, and then the tasks endpoint within the workspace. The endpoints addressing a specific and existing run task association require a run task id, while Associate and List do not.

Using the TFE Provider

The TFE Provider is a very powerful and logical way of managing the overall HCP Terraform configuration and provisioning platform. Standard Terraform code and workspaces are used to describe and provision the necessary resources. The TFE Provider includes the resources to manage run tasks at the same level as using the UI or API, but with all the added benefits of infrastructure as code. As a HashiCorp written and supported provider, the documentation containing the specifics of how to use the resources is located in the public registry:

tfe provider

tfe_organization_run_task

tfe_workspace_run_task

If this mechanism of managing the platform is already in general use, you may be able to just add these resources into the current mix by adding to the existing code/modules. If not, or if the preference is to keep such components isolated, a new workspace or set or workspaces could be provisioned with their own code repositories where the resources could be maintained.

Best practices for Run Tasks

The primary use case for run tasks is to integrate third-party tools and services (including internally developed services). As such, they should be handled in the same ways that you select and implement any software into your environments. They are similar to Terraform Providers, in that the sources of run tasks, and the tools they integrate with, should be carefully scrutinized and evaluated.

Run tasks listed in the registry are provided by HashiCorp and HashiCorp Partners, and can generally be trusted. You still need to do proper due diligence. Run tasks written by others and shared with you should be considered and treated like any OSS software, with more stringent evaluation and handling. For example, you should possibly fork and own the code as opposed to simply downloading and replacing as new changes are introduced.

Security considerations when using Run Tasks

Run tasks, by definition, are invoked through https calls to API endpoints. All normal precautions should be taken to first ensure the two-way handshake between the HCP Terraform service and the run task provider is possible, and then that the opening of the path has been done in a correct manner that does not open the door to any unwanted access. This will involve having the necessary firewall and security groups configured specifically for this exchange.

The run task implementation readily provides for the use of a security token in the form of a signature generated using an HMAC key that can be exchanged between the target run task configuration and the HCP Terraform run task configuration. This is a common security practice that greatly enhances ensuring that only approved and registered access between the two endpoints can take place. This capability should be taken advantage of whenever possible. If a vendor or other run task supplier has not implemented this capability, we recommend having this security token exchange added as an enhancement.

Tips for effective use of Run Tasks in HCP Terraform

Run tasks greatly expand the functionality and capabilities of a Terraform run workflow. The use of run tasks is highly encouraged as the mechanism to meet additional requirements or add additional desired tooling in order to further customize your workflow for your needs. Here are some recommendations on how to use them effectively, both from viewpoints of the benefits and the possible downsides.

Select Run Tasks to integrate based on need and benefit

Take advantage of run tasks, but do not unnecessarily complicate and burden your Terraform workflows by adding several trivial run tasks that do not add additional value. Items like compliance, code quality, standards adherence should certainly be considered. Adding a lot of internal minor tasks may be better handled via other mechanisms.

Manage the number and impact of Run Tasks

Every run task associated with the workflow for a workspace will add some overhead to the run time and possibly resource requirements of every run. This continues the consideration of numbers and need, but also applies to individual run tasks. Run tasks should be quick in-and-out events that capture data and make decisions on it. Extended processing at the time of a run task invocation will cost in time a run will be active while occupying a slot that another run could be using.

Consider the availability of Run Tasks when selecting tools

Similar to the pattern that has developed for Terraform Providers, many of the vendors in the infrastructure related space are supplying run tasks to allow for the integration. It is a benefit to their product offerings and their customers who are already using or considering HashiCorp products. It is in your best interest to look for this integration or intent in products you are using or considering.

Custom Run Task Integrations

Building custom Run Task integrations

In addition to using existing technology partner’s integrations, HCP Terraform customers and other interested parties can build their own custom run task integrations. Custom integrations have access to plan and apply details in the same way as pre-existing run tasks such as from the public registry. They can perform distinct tasks with that data and display custom messages within the run pipeline as well as prevent a run from continuing to the next stage.

There are many useful and valid applications of run tasks that will enhance the Terraform workflow for the entire organization or specialized to meet needs of sections such as application teams. The first place to look for run tasks that may be beneficial to your organization is in the public registry where HashiCorp and HashiCorp technology partners publish run tasks that integrate with commercial and other product offerings. Similar to using Terraform providers, your first choice of selecting and using a run task integration would be to take advantage of the one written for and by the product supplier.

When there is not an available run task to meet the specific needs or support the products and services in use, particularly internal to the organization, developing a run task is the next option. The process and activities to develop a custom run task are fairly straightforward as there is a well defined API that is the core of the integration, and several examples of the run tasks already in existence.

The primary consideration for developing a custom run task is that it should execute short-lived tasks that fit well as an event in the Terraform workflow. Run tasks that are associated with many workspaces will add some run time and overhead to each run, so you want to be careful of adding any considerable load to the platform. The run task should also be accomplishing a task that needs to be and/or is best achieved by being executed inline within the Terraform run workflow.

A primary example of custom run tasks that may be developed would be to integrate with internally built tools and services. Such as when there is a custom code scanning process, a run task could be developed that would invoke the scan and prevent the run from continuing if any corporate compliance controls were discovered that should not be followed. Or if there is a form of an inventory or change tracking process in use, a custom run task would be notified of a run event and use the details to record the activity.

Requirements for building a custom integration

To build a custom integration, you must have an online process capable of receiving API requests from HCP Terraform and responding with a properly formed response containing the status and other pertinent information to a supplied callback URL. When creating/configuring a run task in an organization, the requester supplies an endpoint url that will be available to receive requests from HCP Terraform. A test POST request will be sent to the supplied URL and the run task must respond with a success code 200 for the run task to be created. That is how you will ensure that a run task and all the plumbing to communicate with it has been properly deployed.

The payload supplied to the endpoint will contain a url pointing to the plan or apply data in a JSON output. The integrated service must be able to interpret and process the information provided in order to perform its specific actions.

More details on developing the integration required for a custom run task can be found here: Run Tasks Integration

Since the run task interface is through standard API exchange, any programming language can be used that supports this. Hosting the API endpoint can also be done with any mechanism supporting such services, and will be your responsibility. This template can be used to get started on developing a custom run task using golang: terraform-run-task-scaffolding-go

The Run Tasks integration API

The exchange between HCP Terraform and the run task is defined by the run tasks integration API, documented here: Run Tasks Integration - API Docs - HCP Terraform. Your custom run tasks must fully support and be able to interact with this API as prescribed. It is the contract between HCP Terraform and any run task.

Run Task flow of events

• HCP Terraform will build and populate the initial request to start the run task that has been configured and issue it to the integration endpoint. The run task integration that is listening for and responding to such requests is responsible to acknowledge the request and return a series of responses until a final response that reflects the final status.
• In the response body, there is the ability to supply the current status, that could be “running” for an interim progressive response, or must be “passed” or “failed” for the final response. Basic status description/messages can and should be included.
• At any time, an “outcomes” block can be included in the payload. However, it is a good practice to only send outcomes as part of the progressive status when a run task is running. Run task outcomes can contain tags named "severity" or "status" which will then enrich the outcomes displayed in HCP Terraform, enabling a quicker response to issues containing severity and status.

The suggested sequence of the exchange is as follows:

The initial request will contain an access-token value that is used as the authentication mechanism between HCP Terraform and this instance of the run task invocation. HCP Terraform expects you to callback to the supplied task_result_callback_url using the access_token as an Authentication Header. HCP Terraform expects to receive the callback within 10 minutes, or the task will be considered to have errored.

Sample run task flow transactions and sequences are located in the API documentation.

Best practices for building and testing Custom Run Task integrations

Avoid long-running Run Tasks

As mentioned previously, run tasks should not take a lot of time as that will increase the overall time a Terraform run will take and will be multiplied by how many workspaces are associated and how often runs occur. If a lot of processing is needed for an activity, an alternative would be to use the run task to store enough information for the event to allow another off-cycle process to come in later and do the heavy lifting.

Develop Run Tasks that behave well

Run tasks will potentially execute numerous times a day, often impacting critical infrastructure. The process should be designed to be robust and as straightforward as possible, minimizing the chance of failure. Well-executed run tasks can provide significant benefits, but incorporating a poorly performing process into your workflow can create more issues than it solves.

Develop test suites

Similar to automated infrastructure building, the primary way to test run tasks is with real terraform runs. In line with the best practices for software delivery, it's essential to have repeatable and comprehensive testing for custom run tasks.

To effectively simulate the infrastructure scenarios for run tasks, it may be necessary to create and populate one or more workspaces. These should be designed to yield both successful and failing outcomes from the run tasks, encompassing as wide a range of test cases as possible. Specifically, if the run tasks involve scanning code, a test harness could be developed to generate various code scenarios. This harness would allow the same workspace(s) to be used repeatedly for testing purposes.