Secrets sync

Overview

Vault Enterprise helps organizations manage secrets sprawl by centralizing the governance and control of secrets stored within other secret managers. This functionality, called secrets sync, addresses two main challenges:

• First, some applications or services cannot communicate directly with Vault Enterprise. Cloud providers often support their own secrets management solutions. For example, AWS RDS Proxy and AWS Glue can only access database credentials provided by AWS Secrets Manager. This leads to secrets being spread across different cloud environments and CI/CD systems.

• Second, organizations want full control over how their secrets are stored, organized, accessed, and protected, while also benefiting from cloud-native access. Secrets sync allows application owners to use their existing platform knowledge, reducing the effort needed to adopt Vault. By leveraging cloud provider tools, application owners can maintain their current workflows without learning new technologies.

The following sections are intended for the Vault platform team, which uses Vault Enterprise for secrets lifecycle management. However, their organization requires that the secrets be made available through a cloud-provided secrets manager or CI/CD system.

Secrets sync components

Secrets sync allows organizations to synchronize secrets as needed, continually updating secrets from Vault to external secrets managers so they are always current and accessible by applications or users.

The main components of secrets sync are:

• Sync destinations: Locations where secrets can be synced to. There are five types of destinations:

  • AWS Secrets Manager
  • Azure Key Vault
  • GCP Secret Manager
  • GitHub organization, repository, or environment secrets
  • Vercel Project

• Secret associations: A single Vault secret can be linked to multiple destinations. For example, a Jira token stored as a Vault secret used in CI workflows can be synced to multiple GitHub repositories, each represented by a distinct Vault sync destination. Secret associations link an existing Vault K/V secret to a sync destination. Each secret has its own association, and many associations can share a destination.

• Name templates: You can define the name format of synced secrets for each destination. The default name template is vault/<accessor>/<secret-path>. Name templates help prevent accidental name collisions at the destination and clearly identify the origin of the secrets.

Secret management challenges

Organizations face challenges in deploying and managing secrets, but these challenges also present opportunities to close security gaps and improve efficiency.

People

Managing secrets involves handling sensitive material. Vault’s user interface, API, and CLI allow users to generate, update, delete, rotate, and revoke secrets efficiently and securely.

Accessing sensitive information in new ways or on new machines can lead to accidental data leaks. A centralized secrets management platform like Vault minimizes the risk of breaches caused by human error during manual transfers. Instead of switching between multiple cloud solutions and machines, all secrets are managed, rolled out, and monitored by Vault.

Different teams manage their secret lifecycles differently, writing policies and defining rotation cadences. Syncing secrets from a single management plane increases productivity by eliminating the need to interact with multiple key managers across different clouds.

Processes

Many organizations still use manual processes to rotate secrets, which often require maintenance windows even for minor changes. Secrets that are not regularly rotated provide more time for bad actors to gain access to sensitive data. While Vault secrets sync cannot directly rotate secrets, it can help to automate the process by synchronizing changes done in Vault to external secrets managers. This automation enhances the security posture of cloud-hosted applications by ensuring secrets are always up-to-date.

Secrets consumption sprawl occurs when a single credential is used in many different places. Unrotated secrets and broad internal distribution increase the attack surface. Use Vault to granular manage secret access via paths and keys, segregating them for individual applications and teams.

Managing secrets at scale is tedious, involving endpoint management, consumption, and auditing processes. Centralizing secrets with Vault allows operators to audit in one place, tracking when, by whom, and where secrets are modified or accessed. This centralized approach enhances productivity for governance teams, implements security best practices, and improves compliance monitoring.

Example scenario

Consider this situation: You need to provide credentials, such as database credentials or API keys, to multiple applications provisioned across various cloud providers. These applications already consume secrets from the secrets manager within their respective cloud environments, or the credentials are hard-coded into the application code. Each application development team needs to manage their own secrets while also having them stored centrally for organizational auditing. Each development team is assigned a tenant namespace in your organization's Vault Enterprise deployment.

The recommended organizational layout would be as follows:

• Root administrators
  • Enable the secrets sync feature.
  • Set up tenant namespaces.
• Application A
  • Resides in AWS and GCP.
  • Managed by application development team A.
• Application B
  • Resides in Azure.
  • Managed by application development team B.
• Application development team A
  • Responsible for the development and integration of application A.
  • Namespace users: Can read and update K/V secrets and associate these secrets with a sync destination.
  • Namespace admins: Have elevated admin rights and can set up and configure sync destinations.
• Application development team B
  • Responsible for the development and integration of application B.
  • Namespace users: Can read and update K/V secrets and associate these secrets with a sync destination.
  • Namespace admins: Have elevated admin rights and can set up and configure sync destinations.

This structure allows each development team to manage their secrets independently, while ensuring centralized auditing and control through Vault Enterprise.

Design considerations

Infrastructure architecture

The key architectural aspects to understand are below.

• Network path requirements: Vault requires network access to the secrets sync destinations, and this connectivity must be in place before using the secrets sync feature. The same requirement applies to the disaster recovery (DR) cluster, which also needs outbound network access to the sync destinations. While the DR cluster may not use this connection during normal operations, having it ensures a successful transition in the event of a DR failover.

• Active node processing: Sync operations are only processed on the active node of a primary cluster. Write operations, such as writing an association or a K/V v2 secret, can only occur on the active node. This also means that secrets sync operations will not benefit from scaling the Vault cluster with either performance replication or adding performance standby nodes.

Mount and namespace structure

You can configure multiple secrets sync destinations per Vault namespace, allowing you to manage secrets effectively within your organization. Here’s how you can structure your mounts and namespaces:

• Multiple secrets sync destinations: You can configure several secrets sync destinations within a single namespace. This flexibility enables you to associate secrets from different K/V mounts, as long as they reside within the same namespace as the destination.

• Name templates: Each sync destination has its own name template. If you need different templates for secrets, you can define the same destination multiple times with different names. This allows for customized naming conventions for various secrets.

• Per application configuration: Configuring a secret sync destination for each application enables isolated management of secrets for different applications. This approach helps in managing policy requirements, templating, and rotation separately for each application. It ensures that the secrets of one application do not interfere with those of another.

• One destination per application: It is generally recommended to create one destination per application with multiple secret associations. This practice simplifies the access management via Vault policies, allows effective templating, adheres to rotation requirements for every application, while it can also help with secret segregation at the destination.

• Disaster recovery (DR) considerations: The destinations might need to be DR-aware and have secret-specific configurations. For example, a primary application in AWS us-west-2 region and a DR application in eu-west-1 might need different synced credentials, even if the same application is deployed.

Policy management

The key considerations and best practices for managing policies related to secrets sync are as follows.

• Administrator roles

  • Configuration: Only namespace administrators should configure sync destinations to avoid exposing high-privileged cloud service account credentials.
  • Write capabilities: Namespace administrators, who interact with the sync engine, need high-privileges, such as access to write capabilities for paths under /sys/sync.

• User roles

  • Read access: Users need read access to the Vault secret itself before configuring a sync association. This ensures the Secret Sync feature cannot be used for privilege escalations.
  • Write access: Typically, namespace users have access to the K/V secrets mount to update the value of secrets, which may or may not be synced elsewhere.

• Security considerations

  • Access control: Vault does not control permissions at the destination. It is the platform team's responsibility to configure and maintain proper access controls on external systems to prevent unintentional access to synced secrets.

Policy configuration examples

Here are some example policies for managing secret sync destinations and associations, within a Vault child namespace, e.g. “tenant-1”, while the secrets sync administrator and user will authenticate to the parent namespace “admin”.

Secrets sync administrator policy

# Allow full access to the secrets sync feature.
path "tenant-1/sys/sync/*" {
  capabilities = ["read","create","update","delete","list"]
}
# List existing secret engines.
path "tenant-1/sys/mounts" {
  capabilities = ["read", "list"]
}
# Manage secret engines.
path "tenant-1/sys/mounts/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
# Allow full access to the secret mount called ‘secret’.
path "tenant-1/secret/*" {
  capabilities = ["read","create","update","delete","list","patch"]
}

Secrets sync user policy

# List existing destinations.
path "tenant-1/sys/sync/destinations*" {
  capabilities = ["read", "list"]
}
# List existing associations.
path "tenant-1/sys/sync/associations*" {
  capabilities = ["read", "list"]
}
# Create an association for destination type 'aws-sm’
# and the destination name 'my-account'
path "tenant-1/sys/sync/destinations/aws-sm/my-account/associations/set" {
  capabilities = ["create", "update"]
}
# Remove an association for destination type 'aws-sm’
# and the destination name 'my-account'
path "tenant-1/sys/sync/destinations/aws-sm/my-account/associations/remove" {
  capabilities = ["create", "update"]
}
# Allow full access to the secret mount called ‘secret’.
path "tenant-1/secret/*" {
  capabilities = ["read", "create", "update", "delete", "list", "patch"]
}

Workflow examples

Admin workflow: Admins configure a sync destination.

export VAULT_NAMESPACE=admin/tenant-1
vault write sys/sync/destinations/aws-sm/my-account \
  access_key_id="$ACCESS_KEY_ID" \
  secret_access_key="$SECRET_ACCESS_KEY" \
  region="eu-north-1" \
  secret_name_template="vault/{{ .NamespacePath }}{{ .MountPath }}/{{ .SecretPath }}"

User workflow: User configures an association between an existing K/V secret and the destination.

export VAULT_NAMESPACE=admin/tenant-1
# User creates a K/V secret.
vault kv put secret/database/dev \
  api_key="abc" \
  key_id="123"
# User associates the secret with a destination.
vault write sys/sync/destinations/aws-sm/my-account/associations/set \
   mount="secret" \
   secret_name="database/dev"

If the user wishes to unsync a previously synced secret, they can delete the association:

export VAULT_NAMESPACE=admin/tenant-1
vault write sys/sync/destinations/aws-sm/my-account/associations/remove \
   mount="secret" \
   secret_name="database/dev"

Lifecycle management

• Update and removal: If a secret is updated or removed in Vault, the same changes will be reflected in the external systems. Applications should be rotation-aware to handle these updates seamlessly.

• Last-write-wins: Vault operates with a last-write-wins strategy. If a secret with the same name exists at the destination, Vault will overwrite it during synchronization.

• Use templates and tags: Employ name templates and tags to manage secret access at the destination effectively and to prevent accidental overwrites or exposure.

Automation

Scaling the secrets sync feature via automation can be effectively achieved using Terraform.

Resource types

To configure the secrets sync feature via Terraform, you can to utilize several resource types, each corresponding to different destination types:

• vault_secrets_sync_association
• vault_secrets_sync_aws_destination
• vault_secrets_sync_azure_destination
• vault_secrets_sync_config
• vault_secrets_sync_gcp_destination
• vault_secrets_sync_gh_destination
• vault_secrets_sync_github_apps
• vault_secrets_sync_vercel_destination

Example configuration

Below is an example Terraform configuration to set up the secrets sync feature, including the resources:

• Mount resource: Creates a K/V v2 mount at the path “secret”.
• K/V secret resource: Defines a key-value secret named “database/dev”. The resource also includes a lifecycle meta-argument that ignores changes that are done outside of Terraform, that means you can update this secret via Vault API, CLI or UI.
• AWS destination resource: Configures an AWS secrets sync destination with the name “my-account”.
• Sync association resource: Associates the K/V secret with the AWS sync destination.

resource "vault_mount" "tenant_mount" {
  path      = "secret"
  type      = "kv"
  options = {
    version = "2"
  }
}

resource "vault_kv_secret_v2" "tenant_secret" {
  mount     = vault_mount.tenant_mount.path
  name      = "database/dev"
  data_json = jsonencode(
    {}
  )

  lifecycle {
    ignore_changes = [
      data_json
    ]
  }
}

resource "vault_secrets_sync_aws_destination" "aws" {
  name                 = "my-account"
  access_key_id        = var.access_key_id
  secret_access_key    = var.secret_access_key
  region               = "eu-north-1"
  secret_name_template = "vault/{{ .NamespacePath }}{{ .MountPath }}/{{ .SecretPath }}"
  custom_tags = {
    "ochestrator" = "terraform"
  }
}

resource "vault_secrets_sync_association" "aws-test" {
  name        = vault_secrets_sync_aws_destination.aws.name
  type        = vault_secrets_sync_aws_destination.aws.type
  mount       = vault_mount.tenant_mount.path
  secret_name = vault_kv_secret_v2.tenant_secret.name
}

Best practices for sync destinations

Select the tab below for further guidance on a specific sync destination.