You can use Vault’s PKI Secrets Engine to generate and renew dynamic TLS X.509 certificates. Aside from being useful to secure Consul’s RPC and Raft communication, the PKI secrets engine provides support serving as a Certificate Authority for Consul service mesh.
One of the benefits of using a service mesh is creating, generating, and rotating TLS certificates for services in the service mesh. Using Vault as the CA for your service mesh provides unified management for TLS certificates.
This tutorial configures a Consul datacenter to use Vault as the Consul service mesh CA for new and existing Consul datacenters.
Prerequisites
- vault v0.10.3+
- consul v1.8.0+
- Vault: the tutorial assumes you already have a running Vault 0.10.3+ cluster in your network. You can use a local Vault dev agent or an existing Vault deployment.
Configure Vault ACL
To interact with the PKI Secrets Engine endpoints, you have to generate a Vault token, giving the appropriate permissions to Consul to generate the certificates.
Create Vault policies
The following Vault policies apply to Consul 1.12 and later. For use with earlier Consul versions, refer to the Vault CA provider documentation and select your version from the version dropdown.
Save the configuration in a file named vault-policy-connect-ca.hcl and create the policy with the Vault CLI.
Create Vault token
Once the policy is created, create a Vault token to use for the integration.
Setup your environment
If you use a remote deployment, you can configure a local Consul binary to interact with the datacenter. Set the CONSUL_HTTP_ADDR variable on your local machine or jump host to the IP address of a server.
The same applies to the ACL token to access Consul data. You can export the token with the CONSUL_HTTP_TOKEN variable.
Configure Consul service mesh CA
Once Vault is configured and the necessary policies and tokens are created, configure Consul to use Vault PKI secrets engine as Consul service mesh CA.
Verify Consul service mesh CA configuration
Verify the CA settings by querying Consul for the CA configuration using either the Consul CLI or the REST API.
Configure Consul clients
Once you configured Consul service mesh for the servers, you can configure the clients.
The configuration for the clients does not require to specify the provider but only to enable the service mesh feature.
Verify certificates
Once Vault is configured to act as CA for Consul service mesh you can verify the certificates being used in your service mesh.
Check root and intermediate certificates
Use the /agent/connect/ca/roots endpoint to retrieve the list of currently trusted root certificates.
When a datacenter is initialized, this will only list one trusted root. As in the example above, multiple roots may appear as part of certificate rotation or if the CA got migrated from Consul to Vault.
Check leaf certificates for services
Use the /agent/connect/ca/leaf/<service> endpoint to retrieve the leaf certificates for the Consul service mesh services.
For example, to retrieve the certificate for the service web you can use the following.
Certificate rotation and API caching
The /agent/connect/ca/leaf/<service> endpoint has a caching mechanism that might return stale results for service certificates. You can verify the result’s age by inspecting the header data using the -verbose option with the curl command.
You can verify that the result is served by the cache using the x-cache header that, in this example shows a value of HIT.
In case you notice that the age header is bigger than the value defined for leaf_cert_ttl you can invalidate the cache by pointing at a different Consul node.
Change the CONSUL_HTTP_ADDR variable on your local machine or jump host to the IP address of a different server.
Re-issue the curl command to query the API endpoint. Leave the -verbose parameter set so to verify the cache headers.
This time you can verify that the x-cache value is MISS, meaning that the result is not coming from the cache, and you can verify from the output that the certificates returned are different from the ones returned by the cached request.
Certificate rotation in logs
You can verify the rotation is successfully happening also from Consul logs.
Server agents
When intermediate certificates for the service mesh get rotated, you will get that signaled on the leader’s log.
Client agents
Client agents will get log lines for the rotation of intermediate certificates and the rotation of the service leaf certificate. The log level for those events are DEBUG, so you have to set up that verbosity level to troubleshoot the certificate rotation.
-
Intermediate certificate rotation:
-
Leaf certificate rotation:
Next Steps
In this tutorial, you configured Consul service mesh’s certification Authority using the PKI secrets engine in Vault. You configured a root certificate for Consul service mesh backed by Vault, and generated leaf certificates for services deployed to Consul, signed by this root certificate.
Continue learning about the options of securing Consul by taking additional tutorials on Learn. The Secure Consul Gossip Communication with Vault tutorial teaches you to secure gossip protocol communication inside your Consul clusters. The Generate mTLS Certificates for Consul using Vault tutorial helps you implement mutual TLS for the overall Consul cluster.
Learn more about ACL token administration on Vault with the Generate Consul Tokens with HashiCorp Vault tutorial.
