People & process

HashiCorp Consul is a service networking solution that enables teams to manage secure network connectivity between services and across on-prem and multi-cloud environments and runtimes. Consul offers service discovery, service mesh, traffic management, and automated updates to network infrastructure devices. Integrating these use cases into a common framework often requires different personas to build, maintain and use them.

Operating a networking platform, like Consul for service discovery, requires coordination between different teams to:

• Setup and maintain the platform. This may include the following:
  • Manage access to the platform and defining policies and roles
  • Monitor the platform and the services
  • Manage infrastructure and platform administration
  • Automate (or script) these tasks with the platform API (Consul API)
• Use the platform to register services. This may include the following:
  • Register application services
  • Design and implement health checks for those services

This section explains the Cloud Operating Model (COM), a validated framework that enables teams to operate and succeed in the Cloud. In particular, it will describe the teams and personas responsible for driving Consul adoption and how they interact with each other.

Cloud operating model

The cloud demands a new operating model. Datacenter primitives are changing: static to dynamic infrastructure, security and networking policies based on identity rather than IP addresses, and an automated self-service platform as opposed to manual ticketing systems. While this transformation speeds development and shrinks go-to-market times, it renders much of the previous generation of tools and processes obsolete. To fully realize the benefits of cloud computing, enterprises must transition to scalable dynamic workflows and management at every cloud layer. From this they can deploy shared services enabled by platform teams to improve speed, increase efficiency, and decrease risk.

The Cloud Operating Model (COM) is a new approach for IT operations that organizations need to use to be successful with cloud adoption and thrive in an era of multi-cloud architectures. This overview breaks down the components of that approach and the path to industrializing application delivery across all layers needed to support a cloud-based architecture, articulating the needed changes to people, processes, and tools.

The COM uses a shared service model to organize and deliver technology services, resources, or capabilities to different parts of an organization. This approach allows different teams or business units within the organization to leverage the same set of services, reducing redundancy and promoting consistency.

There are two key roles in the COM:

1. Producers ensure that the shared services are readily accessible and efficiently utilized by the Consumers, empowering them to leverage the benefits of the Consul platform effectively. This is usually the platform team. Some of their responsibilities include:

• Providing a seamless onboarding process to the network automation platform, which is Consul
• Supporting the security access to Consul by configuring the right authentication method integrations and standard default policy implementations
• Providing a multi-tenant and multi-region framework to discover and connect services by peering different Consul Datacenters and/or Partitions
• Offering on-demand processes to deploy self-service Partitions and their services
• Supporting and maintaining the documents and process to integrate with external development tools
• Offering necessary enablement to the consumers.

1. Consumers play a crucial role in leveraging the shared services to meet their specific needs and requirements while adhering to the guidelines and standards set by the Producers. Consumers refer to any team within the organization that uses the shared services provided by the platform team (Producers). This is usually the application development (AppDev) team. Some of their responsibilities include:

• Requesting access to deploy new custom policies for their tenants (or Consul Admin Partition)
• Deploying and configuring services and the health checking process
• Monitoring services health and data provided by the Consul service discovery platform
• Implementing services queries by using Consul DNS.

The platform team is responsible for delivering each layer of a cloud operating model as a standardized shared service that can be consumed by end-users in the organizations.

In addition to providing standardized shared services, the platform team:

• Establishes best practices for developers and practitioners to engage with cloud services. These best practices are then codified directly into workflows and educational programs such as Cloud Centers of Excellence (CCoEs).
• Serves as a single point of integration for other corporate teams, including security, compliance, financial controls, etc. to ensure that their best practices are baked into relevant workflows.
• Provides a central system for tracking, reporting, and auditing to inform adoption and usage of the cloud operating model. This informs progress of the organization and highlights areas where there may be risk exposure: such as security, compliance, and spending.

Organizational structure

Each organization has its own team and role management workflow, depending on their development lifecycle and business alignment. This section provides example teams and personas that fulfill the Cloud Operating Model’s Producers and Consumers roles.

Organizations may call the team and personas described in this section different names. Focus on the infrastructure, development, networking or security activities of each team and personas rather than their specific names.

In addition, this section includes team and personas that are not responsible for operating Consul. However, they are part of the self-service workflows described in the next section.

Teams

The following teams are part of the shared service model:

• Platform team: Platform engineers belong to this team. They operate the services networking platform and ensure that consumers can configure and deploy their required artifacts. It can be divided into different "subteams":
  • Service discovery specialists: There may be experts who focus specifically on implementing and maintaining service discovery mechanisms
  • Traffic management experts: They are not part of the service discovery tasks, but they will need service discovery outcomes for traffic management tasks (load balancing, routing, etc.)
• DevOps team: Usually a cross-functional team responsible for the CI/CD framework and its integration with the discovery catalog and dns routing. Sometimes they could be part of the Platform Team, represented by DevOps engineers
• Networking team: Common task is to design and support the network architecture, working closely with Service Discovery Specialists
• Security team: They implement security measures for service communication and they ensure that any security feature or access is aligned with the organization's security policies

Personas

Within these teams, the following personas are part of the shared service model:

• Developers write and maintain code for the service. They may also register services to Consul, so the services can discover and connect to other services in Consul.
• Platform engineers are cross-discipline and can be represented by different subroles:
  • Infrastructure specialists are responsible for deploying and maintaining the Consul control plane components and provisioning the underlying service discovery platform.
  • DevOps engineers are responsible for the CI/CD setup and automating the Consul deployment.
  • Security architects define and implement the security framework for Consul.
  • SREs are responsible for monitoring and troubleshooting issues related to services health and the platform.
• Security engineers are responsible for implementing security policies and managing access controls, related to service registrations made by developers (e.g. they can define the access policies for different tenants, that can be namespaces or admin partitions, when registering services).
• Network operators ensure that the network is up and running and monitor services health provided by Consul service discovery platform to detect any network related issue. They also could be in charge of reviewing service registration for developers (e.g. ports and traffic are compliant with networking definitions) or DNS integrations.
• Cloud engineers (infrastructure engineers) manage automation deployment to deploy application and networking infrastructure, like load balancers, VPCs, firewalls or any infrastructure required to run applications.

Self service workflows

A self service workflow for operating and using Consul as a service discovery platform requires different steps and stages that enable teams to register and manage their services within Consul. Producers (often platform teams) provide the required platform, enabling downstream consumers (often AppDev teams) to autonomously register and manage the services that will be part of a centralized service catalog.

During the deployment phase of the application lifecycle it is important that services registration is integrated with the development lifecycle, providing a seamless process to manage a service catalog and health checking platform for the services.

The self service workflow should include the following steps, tasks or configurations:

• User accessibility: As a common practice, a self service portal would provide identity or token access methods with the required credentials. These will be controlled by the Consul ACL and auth methods to authorize services access
• Service registration: Any consumer of the Consul platform should be able to request a registration task by providing the required information, like service name, port, tags and health checking settings. The task will follow a configuration as code approach, where the service registration information is part of the development process, making it easier to integrate with any development or orchestration platform
• Validation and triggered events: Consumers will be able to create different scripts or events as part of the service monitoring and health checking process. Consul watches can be configured as part of required dynamic configuration for the applications deployed
• Monitoring and alerting: By using Consul metrics and audit logs, producers will be able to locate potential infrastructure, process or security issues. Consumers can be notified on any specific change or non-compliant updates. These metrics should be integrated with dashboarding tools that help both Producers and Consumers of the platform to identify any change or issue that could impact the application services deployment
• Service catalog retrieval: Consumers, like developers or network operators, will retrieve Consul services’ registered information to integrate with development workflows and other networking operations like network devices configuration integration

From a self-service platform perspective, different roles participate in the workflow:

1. The platform team is responsible for maintaining the shared Consul control plane, the peering settings, and exported services configuration that are required for cross Consul data center workloads.
2. Infrastructure operators and architects deploy and configure the isolated Consul admin partitions depending on the specific team or business unit.
3. Developers, network engineers, or operators will register the service.
   • Developers register services to Consul when they deploy them, using a specific automated process or platform (e.g. Kubernetes integration, Nomad jobs or configuration as code service definition). In addition, they should configure health checks when registering the service.
   • Network engineers will use the services registration information as Consumers to configure load balancers and enable communication within the organization.
   • Security engineers or operators will establish the required security compliance for the health of services.
4. Monitoring services, logs and metrics is a critical task for developers, infrastructure operators and security engineers. This will help them manage and ensure the required availability of services and infrastructure resources, and give some helpful insights about access control (e.g. blocked ACLs).
5. With a centralized service catalog, network operators can use Consul DNS and service discovery to configure load balancers or firewall rules. Developers can use Consul DNS to discover and connect services together in their environments.