Deep Dive: Mutual TLS (mTLS) Authentication
What is Mutual TLS?
Mutual TLS (mTLS) is an extension of the TLS protocol where both the client and server authenticate each other using X.509 certificates during the TLS handshake. Standard TLS only authenticates the server to the client (your browser verifies that api.example.com is who it claims to be). mTLS adds the other direction: the server also verifies the client’s certificate, creating bidirectional cryptographic authentication before a single byte of application data is exchanged.
In a microservices context, mTLS solves a fundamental problem: how does payment-service know that a request claiming to come from order-service actually came from order-service and not from a compromised internal process, a misconfigured service, or a lateral movement attack? With mTLS, the answer is cryptographic — the client presents a certificate signed by a CA that both parties trust. No valid certificate, no connection.
Core Principles
- Transport-layer authentication: Happens at the TLS handshake level, before application code sees the request.
- Certificate-based identity: Identity is a signed X.509 certificate from a trusted CA, not a shared secret.
- Mutual verification: Both sides verify the other’s certificate chain back to a trusted root CA.
- No secrets in headers: Authentication doesn’t add latency to requests after the initial connection — TLS session resumption amortizes the handshake cost.
- Forward secrecy: Ephemeral key exchange (ECDHE) ensures past communications can’t be decrypted even if long-term keys are later compromised.
Authentication Flow
sequenceDiagram
participant C as Client Service
participant S as Server Service
participant CA as Certificate Authority
Note over CA: CA has issued certs to both services
C->>S: ClientHello (TLS 1.3)
S-->>C: ServerHello + Server Certificate
C->>C: Verify server cert chain → CA
C-->>S: Client Certificate + CertificateVerify
S->>S: Verify client cert chain → CA
S->>S: Extract client identity (CN / SAN)
S-->>C: Finished (TLS session established)
C->>S: HTTP request (encrypted)
S-->>C: HTTP response (encrypted)
Part 1: Understanding X.509 Certificates
An X.509 certificate is a digital document that binds a public key to an identity, signed by a Certificate Authority. For service identity, the relevant fields are:
Subject: CN=order-service, O=MyCompany, C=US
Subject Alternative Names:
DNS: order-service.production.svc.cluster.local
URI: spiffe://mycompany.com/production/order-service
Issuer: CN=MyCompany Internal CA
Validity: Not Before 2025-04-22, Not After 2025-07-21 (90 days)
Key Usage: Digital Signature, Key Encipherment
Extended Key Usage: TLS Web Client Authentication, TLS Web Server Authentication
For service-to-service mTLS, the Subject Alternative Name (SAN) is the definitive identity field. Services should be identified by:
- DNS SANs:
order-service.production.svc.cluster.local(stable Kubernetes DNS name) - URI SANs (SPIFFE):
spiffe://trust-domain/service-name(portable cross-cluster identity)
The Common Name (CN) field is deprecated for identity purposes in modern TLS but still widely used.
Certificate Chain
A certificate chain allows services to trust certificates from a common root without distributing every certificate individually:
Root CA (self-signed, offline storage)
└── Intermediate CA (online, issues service certs)
├── order-service.crt (signed by Intermediate CA)
├── payment-service.crt (signed by Intermediate CA)
└── inventory-service.crt (signed by Intermediate CA)
Services only need to trust the Root CA. Any certificate signed by the Intermediate CA (which is in turn signed by the Root CA) is automatically trusted.
Part 2: Setting Up a Private CA
For development and staging, you can create your own CA using OpenSSL or cfssl. For production, use Vault PKI or cert-manager.
OpenSSL CA Setup
#!/bin/bash
# setup-ca.sh — Create a private CA for development
# 1. Create Root CA
mkdir -p ca/{root,intermediate,certs}
# Generate Root CA private key (keep this OFFLINE in production)
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
-out ca/root/root-ca.key
# Create Root CA certificate (10-year validity for root)
openssl req -new -x509 -days 3650 -sha256 \
-key ca/root/root-ca.key \
-out ca/root/root-ca.crt \
-subj "/CN=MyCompany Root CA/O=MyCompany/C=US"
# 2. Create Intermediate CA
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
-out ca/intermediate/intermediate-ca.key
# Create CSR for Intermediate CA
openssl req -new -sha256 \
-key ca/intermediate/intermediate-ca.key \
-out ca/intermediate/intermediate-ca.csr \
-subj "/CN=MyCompany Intermediate CA/O=MyCompany/C=US"
# Sign Intermediate CA with Root CA
cat > ca/intermediate/ext.cnf <<EOF
[v3_ca]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
EOF
openssl x509 -req -days 1825 -sha256 \
-in ca/intermediate/intermediate-ca.csr \
-CA ca/root/root-ca.crt \
-CAkey ca/root/root-ca.key \
-CAcreateserial \
-extfile ca/intermediate/ext.cnf \
-extensions v3_ca \
-out ca/intermediate/intermediate-ca.crt
# Create CA bundle (intermediate + root — services present this chain)
cat ca/intermediate/intermediate-ca.crt ca/root/root-ca.crt \
> ca/intermediate/ca-bundle.crt
echo "CA setup complete."
echo "Root CA: ca/root/root-ca.crt"
echo "Intermediate CA bundle: ca/intermediate/ca-bundle.crt"
Issuing Service Certificates
#!/bin/bash
# issue-cert.sh <service-name> <dns-san> <uri-san>
SERVICE_NAME=$1
DNS_SAN=$2
URI_SAN=$3
mkdir -p ca/certs/${SERVICE_NAME}
# Generate service private key
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
-out ca/certs/${SERVICE_NAME}/${SERVICE_NAME}.key
# Create CSR with SAN extension
cat > /tmp/${SERVICE_NAME}-csr.cnf <<EOF
[req]
req_extensions = v3_req
distinguished_name = dn
[dn]
CN=${SERVICE_NAME}
O=MyCompany
C=US
[v3_req]
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${DNS_SAN}
URI.1 = ${URI_SAN}
EOF
openssl req -new -sha256 \
-key ca/certs/${SERVICE_NAME}/${SERVICE_NAME}.key \
-out ca/certs/${SERVICE_NAME}/${SERVICE_NAME}.csr \
-config /tmp/${SERVICE_NAME}-csr.cnf
# Sign with Intermediate CA (90-day validity)
cat > /tmp/${SERVICE_NAME}-ext.cnf <<EOF
subjectAltName = DNS:${DNS_SAN}, URI:${URI_SAN}
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOF
openssl x509 -req -days 90 -sha256 \
-in ca/certs/${SERVICE_NAME}/${SERVICE_NAME}.csr \
-CA ca/intermediate/intermediate-ca.crt \
-CAkey ca/intermediate/intermediate-ca.key \
-CAcreateserial \
-extfile /tmp/${SERVICE_NAME}-ext.cnf \
-out ca/certs/${SERVICE_NAME}/${SERVICE_NAME}.crt
echo "Certificate issued for ${SERVICE_NAME}"
echo "Cert: ca/certs/${SERVICE_NAME}/${SERVICE_NAME}.crt"
echo "Key: ca/certs/${SERVICE_NAME}/${SERVICE_NAME}.key"
# Issue certificates for our services
./issue-cert.sh order-service \
"order-service.production.svc.cluster.local" \
"spiffe://mycompany.com/production/order-service"
./issue-cert.sh payment-service \
"payment-service.production.svc.cluster.local" \
"spiffe://mycompany.com/production/payment-service"
Part 3: Node.js mTLS Server and Client
mTLS Server (Express)
import https from 'https';
import fs from 'fs';
import express from 'express';
import tls from 'tls';
const app = express();
app.use(express.json());
// ============================================================================
// Certificate Loading
// ============================================================================
const serverOptions: https.ServerOptions = {
// Server's own certificate and key
cert: fs.readFileSync('./certs/payment-service/payment-service.crt'),
key: fs.readFileSync('./certs/payment-service/payment-service.key'),
// CA bundle — only connections from clients with certs signed by this CA
ca: fs.readFileSync('./ca/root/root-ca.crt'),
// REQUIRE client certificate — reject connections without one
requestCert: true,
rejectUnauthorized: true,
// TLS 1.3 only (TLS 1.2 minimum for legacy)
minVersion: 'TLSv1.3',
};
// ============================================================================
// Client Identity Extraction Middleware
// ============================================================================
function extractClientIdentity(req: express.Request): string | null {
const socket = req.socket as tls.TLSSocket;
const cert = socket.getPeerCertificate(true);
if (!cert || !cert.subject) {
return null;
}
// Extract SPIFFE URI SAN if present
const altNames = (cert as any).subjectaltname || '';
const spiffeMatch = altNames.match(/URI:spiffe:\/\/[^,]+/);
if (spiffeMatch) {
return spiffeMatch[0].replace('URI:', '');
}
// Fall back to Common Name
return cert.subject.CN || null;
}
// ============================================================================
// Authentication Middleware
// ============================================================================
const ALLOWED_SERVICES = new Set([
'spiffe://mycompany.com/production/order-service',
'spiffe://mycompany.com/production/inventory-service',
]);
function mtlsAuthMiddleware(
req: express.Request,
res: express.Response,
next: express.NextFunction
) {
const socket = req.socket as tls.TLSSocket;
// Verify TLS client certificate was presented and verified
if (!socket.authorized) {
console.error('mTLS: Client certificate rejected', {
authError: socket.authorizationError,
remoteAddress: req.socket.remoteAddress,
});
return res.status(401).json({
error: 'Client certificate required',
detail: socket.authorizationError,
});
}
const clientIdentity = extractClientIdentity(req);
if (!clientIdentity) {
return res.status(401).json({ error: 'Could not extract client identity from certificate' });
}
// Check if this service is allowed
if (!ALLOWED_SERVICES.has(clientIdentity)) {
console.warn('mTLS: Unauthorized service attempted connection', { clientIdentity });
return res.status(403).json({
error: 'Service not authorized',
identity: clientIdentity,
});
}
// Attach identity to request for downstream use
(req as any).serviceIdentity = clientIdentity;
console.info('mTLS: Authenticated request', {
from: clientIdentity,
path: req.path,
method: req.method,
});
next();
}
// ============================================================================
// Routes
// ============================================================================
app.post(
'/api/payments/charge',
mtlsAuthMiddleware,
(req: express.Request, res: express.Response) => {
const callerIdentity = (req as any).serviceIdentity;
res.json({
success: true,
chargeId: `ch_${Date.now()}`,
processedBy: 'payment-service',
requestedBy: callerIdentity,
});
}
);
// Health check (no mTLS required — for load balancer probes)
app.get('/health', (req, res) => {
res.json({ status: 'healthy' });
});
// ============================================================================
// Start Server
// ============================================================================
const server = https.createServer(serverOptions, app);
server.listen(8443, () => {
console.log('Payment service listening on https://0.0.0.0:8443 (mTLS enforced)');
});
// Graceful shutdown
process.on('SIGTERM', () => {
server.close(() => process.exit(0));
});// Spring Boot mTLS Server (Java 17+)
// application.yml configures TLS; middleware extracts identity
// application.yml:
// server:
// port: 8443
// ssl:
// enabled: true
// key-store: classpath:certs/payment-service.p12
// key-store-password: changeit
// key-store-type: PKCS12
// trust-store: classpath:certs/root-ca.p12
// trust-store-password: changeit
// client-auth: need # REQUIRE client certificate
import jakarta.servlet.*;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.stereotype.Component;
import org.springframework.web.bind.annotation.*;
import java.io.IOException;
import java.security.cert.X509Certificate;
import java.util.Arrays;
import java.util.Set;
@SpringBootApplication
public class PaymentServiceApplication {
public static void main(String[] args) {
SpringApplication.run(PaymentServiceApplication.class, args);
}
}
@Component
public class MtlsAuthFilter implements Filter {
private static final Set<String> ALLOWED_SERVICES = Set.of(
"spiffe://mycompany.com/production/order-service",
"spiffe://mycompany.com/production/inventory-service"
);
@Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
throws IOException, ServletException {
var httpReq = (HttpServletRequest) request;
var httpRes = (HttpServletResponse) response;
// Skip health check
if (httpReq.getRequestURI().equals("/health")) {
chain.doFilter(request, response);
return;
}
var certs = (X509Certificate[]) httpReq.getAttribute(
"jakarta.servlet.request.X509Certificate"
);
if (certs == null || certs.length == 0) {
httpRes.sendError(401, "Client certificate required");
return;
}
String identity = extractIdentity(certs[0]);
if (!ALLOWED_SERVICES.contains(identity)) {
httpRes.sendError(403, "Service not authorized: " + identity);
return;
}
httpReq.setAttribute("serviceIdentity", identity);
chain.doFilter(request, response);
}
private String extractIdentity(X509Certificate cert) {
try {
// Check SAN for SPIFFE URI
var sans = cert.getSubjectAlternativeNames();
if (sans != null) {
for (var san : sans) {
if ((Integer) san.get(0) == 6) { // 6 = URI type
String uri = (String) san.get(1);
if (uri.startsWith("spiffe://")) return uri;
}
}
}
} catch (Exception ignored) {}
// Fallback: CN
return cert.getSubjectX500Principal().getName()
.replaceAll(".*CN=([^,]+).*", "$1");
}
}
@RestController
@RequestMapping("/api/payments")
public class PaymentController {
@PostMapping("/charge")
public ChargeResponse charge(HttpServletRequest req, @RequestBody ChargeRequest body) {
var caller = (String) req.getAttribute("serviceIdentity");
return new ChargeResponse(true, "ch_" + System.currentTimeMillis(), "payment-service", caller);
}
@GetMapping("/health")
public java.util.Map<String, String> health() {
return java.util.Map.of("status", "healthy");
}
record ChargeRequest(String orderId, long amountCents, String currency) {}
record ChargeResponse(boolean success, String chargeId, String processedBy, String requestedBy) {}
}# FastAPI mTLS Server (Python 3.11+)
# Run with: uvicorn main:app --ssl-keyfile ./certs/payment-service.key
# --ssl-certfile ./certs/payment-service.crt
# --ssl-ca-certs ./ca/root-ca.crt
# --ssl-cert-reqs 2 (2 = CERT_REQUIRED)
import ssl
from typing import Optional
from cryptography import x509
from cryptography.hazmat.primitives import serialization
from fastapi import FastAPI, Request, HTTPException
from fastapi.responses import JSONResponse
app = FastAPI()
ALLOWED_SERVICES = {
"spiffe://mycompany.com/production/order-service",
"spiffe://mycompany.com/production/inventory-service",
}
def extract_client_identity(request: Request) -> Optional[str]:
"""Extract SPIFFE URI SAN or CN from the client's TLS certificate."""
# With uvicorn + mTLS, the peer cert is available via the transport
transport = request.scope.get("transport")
if not transport:
return None
ssl_object = transport.get_extra_info("ssl_object")
if not ssl_object:
return None
der_cert = ssl_object.getpeercert(binary_form=True)
if not der_cert:
return None
cert = x509.load_der_x509_certificate(der_cert)
# Prefer SPIFFE URI SAN
try:
san_ext = cert.extensions.get_extension_for_class(x509.SubjectAlternativeName)
for uri in san_ext.value.get_values_for_type(x509.UniformResourceIdentifier):
if uri.startswith("spiffe://"):
return uri
except x509.ExtensionNotFound:
pass
# Fallback: CN
for attr in cert.subject:
if attr.oid == x509.oid.NameOID.COMMON_NAME:
return attr.value
return None
async def require_mtls(request: Request) -> str:
identity = extract_client_identity(request)
if not identity:
raise HTTPException(status_code=401, detail="Client certificate required")
if identity not in ALLOWED_SERVICES:
raise HTTPException(status_code=403, detail=f"Service not authorized: {identity}")
return identity
@app.post("/api/payments/charge")
async def charge(request: Request, body: dict):
caller = await require_mtls(request)
import time
return {
"success": True,
"chargeId": f"ch_{int(time.time() * 1000)}",
"processedBy": "payment-service",
"requestedBy": caller,
}
@app.get("/health")
async def health():
return {"status": "healthy"}// ASP.NET Core mTLS Server (.NET 8)
// Program.cs — configure Kestrel for mTLS
using System.Security.Cryptography.X509Certificates;
using Microsoft.AspNetCore.Server.Kestrel.Https;
var builder = WebApplication.CreateBuilder(args);
// Configure Kestrel to require client certificates
builder.WebHost.ConfigureKestrel(options =>
{
options.ListenAnyIP(8443, listenOptions =>
{
listenOptions.UseHttps(httpsOptions =>
{
httpsOptions.ServerCertificate = X509Certificate2.CreateFromPemFile(
"./certs/payment-service.crt",
"./certs/payment-service.key");
// Load CA for client cert verification
var rootCa = X509Certificate2.CreateFromPemFile("./ca/root-ca.crt");
httpsOptions.ClientCertificateMode = ClientCertificateMode.RequireCertificate;
httpsOptions.ClientCertificateValidation = (cert, chain, errors) =>
{
chain!.ChainPolicy.TrustMode = X509ChainTrustMode.CustomRootTrust;
chain.ChainPolicy.CustomTrustStore.Add(rootCa);
return chain.Build(cert);
};
});
});
});
builder.Services.AddCertificateForwarding(opts => { });
var app = builder.Build();
// mTLS identity extraction
string? ExtractServiceIdentity(HttpContext context)
{
var cert = context.Connection.ClientCertificate;
if (cert == null) return null;
// Check SAN for SPIFFE URI
foreach (var ext in cert.Extensions)
{
if (ext.Oid?.Value == "2.5.29.17") // Subject Alternative Name
{
var formatted = ext.Format(true);
var match = System.Text.RegularExpressions.Regex.Match(
formatted, @"URL=spiffe://[^\s]+");
if (match.Success) return match.Value[4..]; // strip "URL="
}
}
// Fallback: CN
return cert.GetNameInfo(X509NameType.SimpleName, false);
}
var allowedServices = new HashSet<string>
{
"spiffe://mycompany.com/production/order-service",
"spiffe://mycompany.com/production/inventory-service",
};
app.MapPost("/api/payments/charge", (HttpContext context, ChargeRequest body) =>
{
var identity = ExtractServiceIdentity(context);
if (identity == null)
return Results.Json(new { error = "Client certificate required" }, statusCode: 401);
if (!allowedServices.Contains(identity))
return Results.Json(new { error = "Service not authorized", identity }, statusCode: 403);
return Results.Ok(new
{
success = true,
chargeId = $"ch_{DateTimeOffset.UtcNow.ToUnixTimeMilliseconds()}",
processedBy = "payment-service",
requestedBy = identity,
});
});
app.MapGet("/health", () => Results.Ok(new { status = "healthy" }));
app.Run();
record ChargeRequest(string OrderId, long AmountCents, string Currency);mTLS Client (Node.js)
import https from 'https';
import fs from 'fs';
// ============================================================================
// mTLS HTTP Client
// ============================================================================
interface MtlsClientConfig {
certPath: string;
keyPath: string;
caPath: string;
serviceName: string;
}
class MtlsHttpClient {
private agent: https.Agent;
private serviceName: string;
constructor(config: MtlsClientConfig) {
this.serviceName = config.serviceName;
// Create persistent HTTPS agent with mTLS credentials
this.agent = new https.Agent({
cert: fs.readFileSync(config.certPath),
key: fs.readFileSync(config.keyPath),
ca: fs.readFileSync(config.caPath),
rejectUnauthorized: true, // Always verify server cert
minVersion: 'TLSv1.3',
keepAlive: true,
keepAliveMsecs: 30_000,
});
}
async post<T>(url: string, body: unknown): Promise<T> {
const payload = JSON.stringify(body);
return new Promise((resolve, reject) => {
const urlObj = new URL(url);
const options: https.RequestOptions = {
hostname: urlObj.hostname,
port: urlObj.port || 443,
path: urlObj.pathname,
method: 'POST',
agent: this.agent,
headers: {
'Content-Type': 'application/json',
'Content-Length': Buffer.byteLength(payload),
'X-Caller-Service': this.serviceName, // Informational — actual auth is via cert
},
timeout: 5000,
};
const req = https.request(options, (res) => {
let data = '';
res.on('data', (chunk) => { data += chunk; });
res.on('end', () => {
if (res.statusCode && res.statusCode >= 400) {
reject(new Error(`HTTP ${res.statusCode}: ${data}`));
} else {
try {
resolve(JSON.parse(data) as T);
} catch {
reject(new Error(`Failed to parse response: ${data}`));
}
}
});
});
req.on('error', reject);
req.on('timeout', () => {
req.destroy();
reject(new Error('Request timed out'));
});
req.write(payload);
req.end();
});
}
}
// ============================================================================
// Usage: Order Service calling Payment Service
// ============================================================================
const paymentClient = new MtlsHttpClient({
certPath: './certs/order-service/order-service.crt',
keyPath: './certs/order-service/order-service.key',
caPath: './ca/root/root-ca.crt',
serviceName: 'order-service',
});
interface ChargeResponse {
success: boolean;
chargeId: string;
processedBy: string;
}
async function chargeCustomer(orderId: string, amountCents: number): Promise<ChargeResponse> {
return paymentClient.post<ChargeResponse>(
'https://payment-service.production.svc.cluster.local:8443/api/payments/charge',
{ orderId, amountCents, currency: 'USD' }
);
}// Java mTLS HTTP Client (Java 17+ with HttpClient)
import javax.net.ssl.*;
import java.io.*;
import java.net.URI;
import java.net.http.*;
import java.nio.file.*;
import java.security.*;
import java.security.cert.*;
import java.time.Duration;
public class MtlsHttpClient {
private final HttpClient httpClient;
private final String serviceName;
public MtlsHttpClient(String certPath, String keyPath, String caPath, String serviceName)
throws Exception {
this.serviceName = serviceName;
// Load client certificate and key into a KeyStore
KeyStore keyStore = KeyStore.getInstance("PKCS12");
// Combine cert + key into PKCS12 (use openssl or java.security APIs)
// For simplicity, load from a pre-built PKCS12 file:
try (InputStream ks = Files.newInputStream(Paths.get(certPath))) {
keyStore.load(ks, "changeit".toCharArray());
}
// Load CA into TrustStore
KeyStore trustStore = KeyStore.getInstance("PKCS12");
trustStore.load(null);
CertificateFactory cf = CertificateFactory.getInstance("X.509");
try (InputStream caIs = Files.newInputStream(Paths.get(caPath))) {
X509Certificate caCert = (X509Certificate) cf.generateCertificate(caIs);
trustStore.setCertificateEntry("root-ca", caCert);
}
KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
kmf.init(keyStore, "changeit".toCharArray());
TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
tmf.init(trustStore);
SSLContext sslContext = SSLContext.getInstance("TLSv1.3");
sslContext.init(kmf.getKeyManagers(), tmf.getTrustManagers(), null);
this.httpClient = HttpClient.newBuilder()
.sslContext(sslContext)
.connectTimeout(Duration.ofSeconds(5))
.build();
}
public <T> T post(String url, Object body, Class<T> responseType) throws Exception {
String json = new com.fasterxml.jackson.databind.ObjectMapper().writeValueAsString(body);
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.POST(HttpRequest.BodyPublishers.ofString(json))
.header("Content-Type", "application/json")
.header("X-Caller-Service", serviceName)
.timeout(Duration.ofSeconds(5))
.build();
HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
if (response.statusCode() >= 400) {
throw new RuntimeException("HTTP " + response.statusCode() + ": " + response.body());
}
return new com.fasterxml.jackson.databind.ObjectMapper()
.readValue(response.body(), responseType);
}
}
// Usage
var client = new MtlsHttpClient(
"./certs/order-service/order-service.p12",
null, // key is in PKCS12
"./ca/root/root-ca.crt",
"order-service"
);
record ChargeRequest(String orderId, long amountCents, String currency) {}
record ChargeResponse(boolean success, String chargeId, String processedBy) {}
ChargeResponse resp = client.post(
"https://payment-service.production.svc.cluster.local:8443/api/payments/charge",
new ChargeRequest("order-123", 5000, "USD"),
ChargeResponse.class
);# Python mTLS HTTP Client (Python 3.11+)
# pip install httpx
import ssl
from dataclasses import dataclass
from typing import TypeVar, Type
import httpx
T = TypeVar("T")
@dataclass
class MtlsClientConfig:
cert_path: str
key_path: str
ca_path: str
service_name: str
class MtlsHttpClient:
def __init__(self, config: MtlsClientConfig):
self.service_name = config.service_name
# Create SSL context with client certificate
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ssl_context.minimum_version = ssl.TLSVersion.TLSv1_3
ssl_context.load_cert_chain(certfile=config.cert_path, keyfile=config.key_path)
ssl_context.load_verify_locations(cafile=config.ca_path)
ssl_context.verify_mode = ssl.CERT_REQUIRED
ssl_context.check_hostname = True
self._client = httpx.Client(
verify=ssl_context,
timeout=5.0,
headers={"X-Caller-Service": self.service_name},
)
def post(self, url: str, body: dict) -> dict:
response = self._client.post(url, json=body)
response.raise_for_status()
return response.json()
def close(self):
self._client.close()
# Usage: Order Service calling Payment Service
payment_client = MtlsHttpClient(MtlsClientConfig(
cert_path="./certs/order-service/order-service.crt",
key_path="./certs/order-service/order-service.key",
ca_path="./ca/root/root-ca.crt",
service_name="order-service",
))
def charge_customer(order_id: str, amount_cents: int) -> dict:
return payment_client.post(
"https://payment-service.production.svc.cluster.local:8443/api/payments/charge",
{"orderId": order_id, "amountCents": amount_cents, "currency": "USD"},
)// C# mTLS HTTP Client (.NET 8)
using System.Net.Security;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using System.Text.Json;
public class MtlsHttpClient
{
private readonly HttpClient _httpClient;
private readonly string _serviceName;
public MtlsHttpClient(string certPath, string keyPath, string caPath, string serviceName)
{
_serviceName = serviceName;
var clientCert = X509Certificate2.CreateFromPemFile(certPath, keyPath);
var rootCa = X509Certificate2.CreateFromPemFile(caPath);
var handler = new HttpClientHandler
{
ClientCertificateOptions = ClientCertificateOption.Manual,
SslProtocols = System.Security.Authentication.SslProtocols.Tls13,
ServerCertificateCustomValidationCallback = (_, cert, chain, errors) =>
{
// Verify against our internal CA
chain!.ChainPolicy.TrustMode = X509ChainTrustMode.CustomRootTrust;
chain.ChainPolicy.CustomTrustStore.Add(rootCa);
return chain.Build(new X509Certificate2(cert!));
},
};
handler.ClientCertificates.Add(clientCert);
_httpClient = new HttpClient(handler)
{
Timeout = TimeSpan.FromSeconds(5),
};
_httpClient.DefaultRequestHeaders.Add("X-Caller-Service", serviceName);
}
public async Task<T> PostAsync<T>(string url, object body)
{
var json = JsonSerializer.Serialize(body);
var content = new StringContent(json, Encoding.UTF8, "application/json");
var response = await _httpClient.PostAsync(url, content);
var data = await response.Content.ReadAsStringAsync();
if (!response.IsSuccessStatusCode)
throw new HttpRequestException($"HTTP {(int)response.StatusCode}: {data}");
return JsonSerializer.Deserialize<T>(data)!;
}
}
// Usage: Order Service calling Payment Service
var paymentClient = new MtlsHttpClient(
certPath: "./certs/order-service/order-service.crt",
keyPath: "./certs/order-service/order-service.key",
caPath: "./ca/root/root-ca.crt",
serviceName: "order-service"
);
record ChargeRequest(string OrderId, long AmountCents, string Currency);
record ChargeResponse(bool Success, string ChargeId, string ProcessedBy);
async Task<ChargeResponse> ChargeCustomer(string orderId, long amountCents)
{
return await paymentClient.PostAsync<ChargeResponse>(
"https://payment-service.production.svc.cluster.local:8443/api/payments/charge",
new ChargeRequest(orderId, amountCents, "USD")
);
}Part 4: Go Implementation
Go’s crypto/tls package makes mTLS straightforward to implement:
package mtls
import (
"crypto/tls"
"crypto/x509"
"fmt"
"net/http"
"os"
)
// ============================================================================
// mTLS Server Configuration
// ============================================================================
func NewMTLSServer(certFile, keyFile, caFile string) (*http.Server, error) {
// Load server certificate and key
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
return nil, fmt.Errorf("load server cert: %w", err)
}
// Load CA certificate pool
caCert, err := os.ReadFile(caFile)
if err != nil {
return nil, fmt.Errorf("read CA cert: %w", err)
}
caCertPool := x509.NewCertPool()
if !caCertPool.AppendCertsFromPEM(caCert) {
return nil, fmt.Errorf("failed to parse CA certificate")
}
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{cert},
ClientCAs: caCertPool,
// RequireAndVerifyClientCert — reject connections without valid client cert
ClientAuth: tls.RequireAndVerifyClientCert,
MinVersion: tls.VersionTLS13,
// Modern cipher suites only
CipherSuites: []uint16{
tls.TLS_AES_128_GCM_SHA256,
tls.TLS_AES_256_GCM_SHA384,
tls.TLS_CHACHA20_POLY1305_SHA256,
},
}
mux := http.NewServeMux()
mux.HandleFunc("/api/payments/charge", withMTLSAuth(handleCharge))
mux.HandleFunc("/health", handleHealth)
return &http.Server{
Addr: ":8443",
Handler: mux,
TLSConfig: tlsConfig,
}, nil
}
// ============================================================================
// Extract Client Identity from Certificate
// ============================================================================
func getClientIdentity(r *http.Request) (string, error) {
if r.TLS == nil || len(r.TLS.PeerCertificates) == 0 {
return "", fmt.Errorf("no client certificate presented")
}
cert := r.TLS.PeerCertificates[0]
// Prefer SPIFFE URI SAN
for _, uri := range cert.URIs {
if uri.Scheme == "spiffe" {
return uri.String(), nil
}
}
// Fall back to DNS SAN
if len(cert.DNSNames) > 0 {
return cert.DNSNames[0], nil
}
// Last resort: Common Name
if cert.Subject.CommonName != "" {
return cert.Subject.CommonName, nil
}
return "", fmt.Errorf("could not determine client identity")
}
// ============================================================================
// mTLS Authorization Middleware
// ============================================================================
var allowedServices = map[string]bool{
"spiffe://mycompany.com/production/order-service": true,
"spiffe://mycompany.com/production/inventory-service": true,
}
func withMTLSAuth(next http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
identity, err := getClientIdentity(r)
if err != nil {
http.Error(w, "Client certificate required", http.StatusUnauthorized)
return
}
if !allowedServices[identity] {
http.Error(w, fmt.Sprintf("Service not authorized: %s", identity), http.StatusForbidden)
return
}
// Inject identity into context for handlers
ctx := WithServiceIdentity(r.Context(), identity)
next(w, r.WithContext(ctx))
}
}
// ============================================================================
// mTLS Client Configuration
// ============================================================================
func NewMTLSClient(certFile, keyFile, caFile string) (*http.Client, error) {
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
return nil, fmt.Errorf("load client cert: %w", err)
}
caCert, err := os.ReadFile(caFile)
if err != nil {
return nil, fmt.Errorf("read CA cert: %w", err)
}
caCertPool := x509.NewCertPool()
if !caCertPool.AppendCertsFromPEM(caCert) {
return nil, fmt.Errorf("failed to parse CA certificate")
}
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{cert},
RootCAs: caCertPool,
MinVersion: tls.VersionTLS13,
InsecureSkipVerify: false, // Never skip verification in production
}
transport := &http.Transport{
TLSClientConfig: tlsConfig,
// Connection pooling for performance
MaxIdleConns: 100,
MaxIdleConnsPerHost: 10,
}
return &http.Client{
Transport: transport,
Timeout: 5_000_000_000, // 5 seconds in nanoseconds
}, nil
}
Part 5: Kubernetes cert-manager Integration
cert-manager automates certificate issuance and rotation in Kubernetes, eliminating manual certificate management entirely.
Install cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true
Create a ClusterIssuer with Your Internal CA
# cluster-issuer.yaml
apiVersion: v1
kind: Secret
metadata:
name: internal-ca-key-pair
namespace: cert-manager
type: kubernetes.io/tls
data:
tls.crt: <base64-encoded intermediate CA cert>
tls.key: <base64-encoded intermediate CA key>
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: internal-ca-issuer
spec:
ca:
secretName: internal-ca-key-pair
Certificate Resource for Each Service
# payment-service-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: payment-service-cert
namespace: production
spec:
secretName: payment-service-tls
duration: 2160h # 90 days
renewBefore: 360h # Renew 15 days before expiry
subject:
organizations:
- MyCompany
commonName: payment-service
dnsNames:
- payment-service
- payment-service.production
- payment-service.production.svc.cluster.local
uris:
- spiffe://mycompany.com/production/payment-service
usages:
- digital signature
- key encipherment
- server auth
- client auth
issuerRef:
name: internal-ca-issuer
kind: ClusterIssuer
Deployment Using the Certificate Secret
# payment-service-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-service
namespace: production
spec:
replicas: 3
selector:
matchLabels:
app: payment-service
template:
metadata:
labels:
app: payment-service
spec:
containers:
- name: payment-service
image: mycompany/payment-service:latest
ports:
- containerPort: 8443
volumeMounts:
- name: tls-certs
mountPath: /app/certs/payment-service
readOnly: true
- name: ca-bundle
mountPath: /app/certs/ca
readOnly: true
env:
- name: TLS_CERT_PATH
value: /app/certs/payment-service/tls.crt
- name: TLS_KEY_PATH
value: /app/certs/payment-service/tls.key
- name: CA_CERT_PATH
value: /app/certs/ca/root-ca.crt
volumes:
- name: tls-certs
secret:
secretName: payment-service-tls
- name: ca-bundle
configMap:
name: internal-ca-bundle
NetworkPolicy to Enforce mTLS-Only Communication
# payment-service-network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: payment-service-allow-mtls-only
namespace: production
spec:
podSelector:
matchLabels:
app: payment-service
policyTypes:
- Ingress
ingress:
# Only allow traffic from order-service and inventory-service on mTLS port
- from:
- podSelector:
matchLabels:
app: order-service
- podSelector:
matchLabels:
app: inventory-service
ports:
- protocol: TCP
port: 8443
Part 6: Certificate Rotation Without Downtime
Certificate rotation is the hardest operational aspect of mTLS. Here’s a safe rotation procedure:
Zero-Downtime Rotation Strategy
The key insight: certificates have a validity window, not a point-in-time validity. During rotation, briefly run with both the old and new certificates loaded simultaneously (dual-serving), then drop the old one.
import fs from 'fs';
import https from 'https';
import tls from 'tls';
// ============================================================================
// Dynamic Certificate Loading with Hot Reload
// ============================================================================
class CertificateManager {
private cert: Buffer;
private key: Buffer;
private ca: Buffer;
private certPath: string;
private keyPath: string;
private caPath: string;
constructor(certPath: string, keyPath: string, caPath: string) {
this.certPath = certPath;
this.keyPath = keyPath;
this.caPath = caPath;
this.cert = fs.readFileSync(certPath);
this.key = fs.readFileSync(keyPath);
this.ca = fs.readFileSync(caPath);
}
reload(): void {
try {
const newCert = fs.readFileSync(this.certPath);
const newKey = fs.readFileSync(this.keyPath);
const newCa = fs.readFileSync(this.caPath);
// Validate that the new cert/key pair is valid before applying
tls.createSecureContext({ cert: newCert, key: newKey });
this.cert = newCert;
this.key = newKey;
this.ca = newCa;
console.info('CertificateManager: Certificates reloaded successfully');
} catch (error) {
console.error('CertificateManager: Failed to reload certificates', error);
// Keep using existing certificates
}
}
getContext(): tls.SecureContextOptions {
return {
cert: this.cert,
key: this.key,
ca: this.ca,
};
}
watchForChanges(intervalMs = 60_000): void {
setInterval(() => this.reload(), intervalMs);
}
}
// ============================================================================
// HTTPS Server with Dynamic SNI Callback
// ============================================================================
const certManager = new CertificateManager(
process.env.TLS_CERT_PATH!,
process.env.TLS_KEY_PATH!,
process.env.CA_CERT_PATH!
);
// Watch for certificate changes every 60 seconds
certManager.watchForChanges(60_000);
const server = https.createServer({
// Use SNICallback to pick up certificate reloads dynamically
SNICallback: (servername, callback) => {
const ctx = tls.createSecureContext(certManager.getContext());
callback(null, ctx);
},
requestCert: true,
rejectUnauthorized: true,
minVersion: 'TLSv1.3',
});// Java hot-reload certificate manager (Spring Boot)
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManagerFactory;
import java.io.FileInputStream;
import java.nio.file.*;
import java.security.KeyStore;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.util.concurrent.atomic.AtomicReference;
@Component
public class CertificateManager {
private final String certPath = System.getenv("TLS_CERT_PATH");
private final String keyPath = System.getenv("TLS_KEY_PATH");
private final String caPath = System.getenv("CA_CERT_PATH");
private final AtomicReference<SSLContext> currentContext = new AtomicReference<>();
public CertificateManager() throws Exception {
currentContext.set(buildSslContext());
}
public SSLContext getSslContext() {
return currentContext.get();
}
@Scheduled(fixedDelay = 60_000) // Check every 60 seconds
public void reload() {
try {
SSLContext newCtx = buildSslContext();
currentContext.set(newCtx);
System.out.println("CertificateManager: Certificates reloaded successfully");
} catch (Exception e) {
System.err.println("CertificateManager: Failed to reload certificates: " + e.getMessage());
// Keep using existing certificates
}
}
private SSLContext buildSslContext() throws Exception {
// Load PKCS12 keystore (cert + key)
KeyStore keyStore = KeyStore.getInstance("PKCS12");
try (FileInputStream fis = new FileInputStream(certPath)) {
keyStore.load(fis, "changeit".toCharArray());
}
// Load CA
KeyStore trustStore = KeyStore.getInstance("PKCS12");
trustStore.load(null);
CertificateFactory cf = CertificateFactory.getInstance("X.509");
try (FileInputStream fis = new FileInputStream(caPath)) {
X509Certificate ca = (X509Certificate) cf.generateCertificate(fis);
trustStore.setCertificateEntry("root-ca", ca);
}
KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
kmf.init(keyStore, "changeit".toCharArray());
TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509");
tmf.init(trustStore);
SSLContext ctx = SSLContext.getInstance("TLSv1.3");
ctx.init(kmf.getKeyManagers(), tmf.getTrustManagers(), null);
return ctx;
}
}# Python hot-reload certificate manager
import ssl
import threading
import time
import logging
from dataclasses import dataclass
from typing import Optional
logger = logging.getLogger(__name__)
@dataclass
class CertPaths:
cert_path: str
key_path: str
ca_path: str
class CertificateManager:
def __init__(self, paths: CertPaths, reload_interval_s: float = 60.0):
self._paths = paths
self._reload_interval = reload_interval_s
self._context: Optional[ssl.SSLContext] = None
self._lock = threading.Lock()
self._load()
self._start_watcher()
def _load(self) -> None:
try:
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ctx.minimum_version = ssl.TLSVersion.TLSv1_3
ctx.load_cert_chain(
certfile=self._paths.cert_path,
keyfile=self._paths.key_path,
)
ctx.load_verify_locations(cafile=self._paths.ca_path)
ctx.verify_mode = ssl.CERT_REQUIRED
with self._lock:
self._context = ctx
logger.info("CertificateManager: Certificates reloaded successfully")
except Exception as e:
logger.error(f"CertificateManager: Failed to reload certificates: {e}")
# Keep using existing context
def get_context(self) -> ssl.SSLContext:
with self._lock:
return self._context # type: ignore
def _start_watcher(self) -> None:
def watcher():
while True:
time.sleep(self._reload_interval)
self._load()
t = threading.Thread(target=watcher, daemon=True)
t.start()
# Usage with uvicorn (pass ssl_context dynamically via a custom server)
import os
cert_manager = CertificateManager(CertPaths(
cert_path=os.environ["TLS_CERT_PATH"],
key_path=os.environ["TLS_KEY_PATH"],
ca_path=os.environ["CA_CERT_PATH"],
))// C# hot-reload certificate manager (.NET 8)
using System.Net.Security;
using System.Security.Cryptography.X509Certificates;
public class CertificateManager : IDisposable
{
private readonly string _certPath;
private readonly string _keyPath;
private readonly string _caPath;
private volatile X509Certificate2 _cert;
private volatile X509Certificate2 _ca;
private readonly Timer _reloadTimer;
private readonly object _lock = new();
public CertificateManager(string certPath, string keyPath, string caPath,
TimeSpan? reloadInterval = null)
{
_certPath = certPath;
_keyPath = keyPath;
_caPath = caPath;
(_cert, _ca) = Load();
// Watch for certificate changes
var interval = reloadInterval ?? TimeSpan.FromSeconds(60);
_reloadTimer = new Timer(_ => Reload(), null, interval, interval);
}
public X509Certificate2 GetCertificate() => _cert;
public X509Certificate2 GetCa() => _ca;
public bool ValidateServerCertificate(
object sender, X509Certificate? cert, X509Chain? chain, SslPolicyErrors errors)
{
if (cert == null) return false;
chain!.ChainPolicy.TrustMode = X509ChainTrustMode.CustomRootTrust;
chain.ChainPolicy.CustomTrustStore.Add(_ca);
return chain.Build(new X509Certificate2(cert));
}
private void Reload()
{
try
{
var (newCert, newCa) = Load();
lock (_lock)
{
_cert = newCert;
_ca = newCa;
}
Console.WriteLine("CertificateManager: Certificates reloaded successfully");
}
catch (Exception ex)
{
Console.Error.WriteLine($"CertificateManager: Failed to reload: {ex.Message}");
// Keep using existing certificates
}
}
private (X509Certificate2 cert, X509Certificate2 ca) Load()
{
var cert = X509Certificate2.CreateFromPemFile(_certPath, _keyPath);
var ca = X509Certificate2.CreateFromPemFile(_caPath);
return (cert, ca);
}
public void Dispose() => _reloadTimer.Dispose();
}
// Usage
var certManager = new CertificateManager(
certPath: Environment.GetEnvironmentVariable("TLS_CERT_PATH")!,
keyPath: Environment.GetEnvironmentVariable("TLS_KEY_PATH")!,
caPath: Environment.GetEnvironmentVariable("CA_CERT_PATH")!
);When cert-manager renews certificates (writing new files to the mounted secret volume), the CertificateManager picks up the new certificate on its next reload interval — no restart required.
Part 7: Production Checklist
Certificate Management
- Use a private CA with separate Root and Intermediate CAs
- Root CA private key is stored offline (HSM or air-gapped server)
- Certificates have 90-day validity or less (shorter = more rotation practice)
- Automated rotation via cert-manager or Vault PKI
- Monitor certificate expiry with alerts at 30 days and 7 days
- Test rotation procedure in staging before production
TLS Configuration
- Enforce TLS 1.3 minimum (drop TLS 1.2 if all services support it)
-
rejectUnauthorized: trueon all clients (noNODE_TLS_REJECT_UNAUTHORIZED=0) - Verify server certificate hostname matches the service DNS name
- Use modern ECDH cipher suites (P-256 or P-384)
- Enable TLS session tickets for performance (reduces handshake overhead)
Identity Verification
- Extract identity from Subject Alternative Name (SAN), not just CN
- Implement allowlist of authorized caller identities per service
- Log all authentication failures with certificate details
- Log successful authentications with client identity for audit trail
Kubernetes-Specific
- cert-manager ClusterIssuer configured with internal CA
- Certificate resources created for every service with correct SANs
- NetworkPolicy restricts mTLS traffic to authorized service-to-service paths
- CA bundle distributed as ConfigMap (not Secret — CA cert is public)
- Secrets rotation does not require pod restart (volume mount + SNI callback)
Operational
- Runbook for emergency certificate revocation
- CRL (Certificate Revocation List) or OCSP Stapling if revocation is needed fast
- Metrics: TLS handshake errors, certificate expiry days
- Alerts: handshake error spike, certificate expiry < 7 days
- Regular fire drills: simulate certificate expiry and rotation
Conclusion
Mutual TLS is the most robust server-to-server authentication mechanism available because it operates at the transport layer, provides mutual verification, and is completely transparent to application code. The cryptographic identity is built into the network connection itself — there’s no token to steal from a header, no API key to extract from an environment variable.
The cost is operational complexity. Certificate lifecycle management, rotation automation, and CA infrastructure are non-trivial to operate. For most teams, the right path is to layer mTLS at the infrastructure level (via a service mesh like Istio) and handle application-level authorization separately.
Start with cert-manager in Kubernetes, keep certificate validity short (90 days or less), automate everything, and treat certificate expiry monitoring as a critical alert. The first time you rotate certificates in production without downtime, the operational investment will feel entirely worthwhile.