Automatic Certificate Rotation with cert-manager and Linkerd

Certificates expire. In a Kubernetes homelab with Linkerd service mesh, this means the identity issuer certificate needs renewal annually. Without automation, this becomes a manual task that’s easy to forget until mTLS breaks across the cluster.

This post covers installing cert-manager on a bare-metal kubeadm cluster and configuring it to automatically rotate Linkerd’s identity issuer certificate.

The Problem

Linkerd uses a two-tier PKI:

Certificate	Purpose	Default Lifetime
Trust Anchor	Root CA for the mesh	10 years
Identity Issuer	Signs proxy certificates	1 year

The identity issuer expires annually. When it does, new proxy sidecars cannot obtain valid certificates, breaking mTLS. The trust anchor rarely needs rotation, but the identity issuer requires attention.

Solution Overview

┌─────────────────────────────────────────────────────────────────┐
│                     cert-manager                                │
│  ┌──────────────────────┐    ┌────────────────────────────────┐ │
│  │ linkerd-trust-anchor │───▶│ linkerd-identity-issuer (Cert) │ │
│  │      (Issuer)        │    │  secretName: ...-certmanager   │ │
│  └──────────────────────┘    └────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
                                          │
                                          ▼
┌─────────────────────────────────────────────────────────────────┐
│                   linkerd-cert-sync CronJob                     │
│  Runs hourly, converts tls.crt/tls.key → crt.pem/key.pem       │
│  Restarts linkerd-identity deployment if certificate changed   │
└─────────────────────────────────────────────────────────────────┘
                                          │
                                          ▼
┌─────────────────────────────────────────────────────────────────┐
│                     linkerd namespace                           │
│  ┌────────────────────────┐    ┌──────────────────────────────┐ │
│  │ linkerd-identity-issuer│    │   linkerd-identity           │ │
│  │  (Opaque Secret)       │◀───│   (Deployment)               │ │
│  │  crt.pem, key.pem      │    │   Uses certificate for mTLS  │ │
│  └────────────────────────┘    └──────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘

cert-manager issues and renews the certificate. A CronJob syncs it to Linkerd’s expected secret format and restarts the identity service when changes occur.

Directory Structure

infrastructure/cert-manager/
├── cert-manager-values.yaml      # Helm configuration
├── cluster-issuers.yaml          # Self-signed and CA issuers
├── linkerd-certificates.yaml     # Linkerd cert + sync CronJob + RBAC
├── prometheus-alerts.yaml        # Expiry alerting rules
├── setup-cert-manager.sh         # Main installation script
├── setup-linkerd-certs.sh        # Linkerd integration script
└── README.md

Installation

Step 1: Install cert-manager

Helm values for cert-manager with Prometheus metrics enabled:

# cert-manager-values.yaml
crds:
  enabled: true
  keep: true

prometheus:
  enabled: true
  servicemonitor:
    enabled: true
    namespace: cert-manager
    labels:
      release: prometheus

resources:
  requests:
    cpu: 10m
    memory: 32Mi
  limits:
    memory: 64Mi

webhook:
  resources:
    requests:
      cpu: 10m
      memory: 32Mi
    limits:
      memory: 64Mi

cainjector:
  resources:
    requests:
      cpu: 10m
      memory: 32Mi
    limits:
      memory: 128Mi

featureGates: "ExperimentalCertificateSigningRequestControllers=true"

global:
  logLevel: 2

Installation script:

#!/bin/bash
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
NAMESPACE="cert-manager"

# Create namespace with Linkerd injection
kubectl create namespace "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
kubectl annotate namespace "$NAMESPACE" linkerd.io/inject=enabled --overwrite

# Add Helm repo
helm repo add jetstack https://charts.jetstack.io 2>/dev/null || true
helm repo update jetstack

# Install cert-manager
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace "$NAMESPACE" \
  --values "$SCRIPT_DIR/cert-manager-values.yaml" \
  --wait

# Wait for webhook
kubectl wait --for=condition=Available deployment/cert-manager-webhook \
  -n "$NAMESPACE" --timeout=120s

# Create ClusterIssuers
kubectl apply -f "$SCRIPT_DIR/cluster-issuers.yaml"

# Create Prometheus alerts
kubectl apply -f "$SCRIPT_DIR/prometheus-alerts.yaml"

# Wait for CA
kubectl wait --for=condition=Ready certificate/selfsigned-ca \
  -n "$NAMESPACE" --timeout=60s

Step 2: Create ClusterIssuers

Two ClusterIssuers provide flexibility for issuing certificates:

---
# Self-signed ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}
---
# Self-signed CA certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: selfsigned-ca
  secretName: selfsigned-ca-secret
  duration: 87600h # 10 years
  renewBefore: 8760h # 1 year
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer
    group: cert-manager.io
---
# CA ClusterIssuer using the self-signed CA
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-issuer
spec:
  ca:
    secretName: selfsigned-ca-secret

Step 3: Configure Linkerd Certificate Rotation

The Linkerd integration requires the trust anchor CA key to be available. This should exist from the initial Linkerd installation:

infrastructure/linkerd/
├── ca.crt    # Trust anchor certificate
├── ca.key    # Trust anchor private key
└── ...

The Linkerd certificate configuration:

---
# Issuer using Linkerd trust anchor
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: linkerd-trust-anchor
  namespace: linkerd
spec:
  ca:
    secretName: linkerd-trust-anchor
---
# Certificate for identity issuer
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linkerd-identity-issuer
  namespace: linkerd
spec:
  secretName: linkerd-identity-issuer-certmanager
  duration: 8760h     # 1 year
  renewBefore: 720h   # 30 days before expiry
  issuerRef:
    name: linkerd-trust-anchor
    kind: Issuer
  commonName: identity.linkerd.cluster.local
  isCA: true
  privateKey:
    algorithm: ECDSA
    size: 256
  usages:
    - cert sign
    - crl sign
    - server auth
    - client auth

The Secret Format Problem

cert-manager creates TLS secrets with keys tls.crt and tls.key. Linkerd expects crt.pem and key.pem. A CronJob handles this translation:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: linkerd-cert-sync
  namespace: linkerd
spec:
  schedule: "0 * * * *"  # Every hour
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: linkerd-cert-sync
          restartPolicy: OnFailure
          containers:
          - name: sync
            image: bitnami/kubectl:latest
            command:
            - /bin/bash
            - -c
            - |
              set -e

              TLS_CRT=$(kubectl get secret linkerd-identity-issuer-certmanager -n linkerd -o jsonpath='{.data.tls\.crt}')
              TLS_KEY=$(kubectl get secret linkerd-identity-issuer-certmanager -n linkerd -o jsonpath='{.data.tls\.key}')
              CURRENT_CRT=$(kubectl get secret linkerd-identity-issuer -n linkerd -o jsonpath='{.data.crt\.pem}' 2>/dev/null || echo "")

              if [ "$TLS_CRT" != "$CURRENT_CRT" ]; then
                echo "Certificate changed, updating linkerd-identity-issuer secret..."

                kubectl create secret generic linkerd-identity-issuer \
                  --from-literal=crt.pem="$(echo $TLS_CRT | base64 -d)" \
                  --from-literal=key.pem="$(echo $TLS_KEY | base64 -d)" \
                  --namespace=linkerd \
                  --dry-run=client -o yaml | kubectl apply -f -

                kubectl label secret linkerd-identity-issuer -n linkerd \
                  linkerd.io/control-plane-component=identity \
                  linkerd.io/control-plane-ns=linkerd \
                  --overwrite

                kubectl rollout restart deployment linkerd-identity -n linkerd
                echo "Linkerd identity issuer updated and identity service restarted"
              else
                echo "Certificate unchanged, no action needed"
              fi

The CronJob requires RBAC to read secrets and restart deployments:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: linkerd-cert-sync
  namespace: linkerd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: linkerd-cert-sync
  namespace: linkerd
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "create", "update", "patch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: linkerd-cert-sync
  namespace: linkerd
subjects:
- kind: ServiceAccount
  name: linkerd-cert-sync
  namespace: linkerd
roleRef:
  kind: Role
  name: linkerd-cert-sync
  apiGroup: rbac.authorization.k8s.io

Step 4: Add Prometheus Alerts

Alert before certificates expire:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cert-manager-alerts
  namespace: cert-manager
  labels:
    release: prometheus
    app: cert-manager
spec:
  groups:
  - name: cert-manager
    rules:
    - alert: CertificateExpiringSoon
      expr: |
        certmanager_certificate_expiration_timestamp_seconds - time() < 30 * 24 * 3600
        and
        certmanager_certificate_expiration_timestamp_seconds - time() > 14 * 24 * 3600
      for: 1h
      labels:
        severity: warning
      annotations:
        summary: "Certificate {{ $labels.name }} expires in less than 30 days"

    - alert: CertificateExpiringVerySoon
      expr: |
        certmanager_certificate_expiration_timestamp_seconds - time() < 14 * 24 * 3600
        and
        certmanager_certificate_expiration_timestamp_seconds - time() > 7 * 24 * 3600
      for: 1h
      labels:
        severity: warning
      annotations:
        summary: "Certificate {{ $labels.name }} expires in less than 14 days"

    - alert: CertificateExpiringCritical
      expr: |
        certmanager_certificate_expiration_timestamp_seconds - time() < 7 * 24 * 3600
      for: 10m
      labels:
        severity: critical
      annotations:
        summary: "Certificate {{ $labels.name }} expires in less than 7 days"

    - alert: CertificateNotReady
      expr: |
        certmanager_certificate_ready_status{condition="False"} == 1
      for: 15m
      labels:
        severity: warning
      annotations:
        summary: "Certificate {{ $labels.name }} is not ready"

    - alert: CertManagerAbsent
      expr: |
        absent(up{job="cert-manager"})
      for: 10m
      labels:
        severity: critical
      annotations:
        summary: "cert-manager is not running"

Running the Installation

# Install cert-manager
cd infrastructure/cert-manager/
./setup-cert-manager.sh

# Configure Linkerd certificate rotation
./setup-linkerd-certs.sh

Verification

Check certificate status:

kubectl get certificates -A

NAMESPACE      NAME                      READY   SECRET                                AGE
cert-manager   selfsigned-ca             True    selfsigned-ca-secret                  10m
linkerd        linkerd-identity-issuer   True    linkerd-identity-issuer-certmanager   5m

Check expiry date:

kubectl get secret linkerd-identity-issuer -n linkerd -o jsonpath='{.data.crt\.pem}' | \
  base64 -d | openssl x509 -noout -dates

notBefore=Jan  1 11:48:02 2026 GMT
notAfter=Jan  1 11:48:02 2027 GMT

Verify Linkerd is healthy:

linkerd check

Manually trigger a sync:

kubectl create job --from=cronjob/linkerd-cert-sync manual-sync -n linkerd
kubectl logs job/manual-sync -n linkerd

What This Does Not Cover

cert-manager handles X.509 certificates. The following require separate tracking:

Token Type	Example	Expiry
GitLab deploy tokens	Registry pull secrets	Configurable in GitLab
AWS access keys	S3 credentials	Optional rotation
API keys	Splunk HEC tokens	Typically no expiry
Database passwords	PostgreSQL credentials	No expiry

These tokens need a different solution such as a ConfigMap-based registry or external secrets management.

Summary

Component	Status
cert-manager	Manages certificate lifecycle
Self-signed CA	10-year internal CA
Linkerd identity issuer	Auto-renews 30 days before expiry
Sync CronJob	Converts secret format hourly
Prometheus alerts	30/14/7 day warnings

The Linkerd identity issuer certificate now renews automatically. Prometheus alerts provide early warning if renewal fails.

Configuration available at k8s-configs/infrastructure/cert-manager.

The Problem#

Solution Overview#

Directory Structure#

Installation#

Step 1: Install cert-manager#

Step 2: Create ClusterIssuers#

Step 3: Configure Linkerd Certificate Rotation#

The Secret Format Problem#

Step 4: Add Prometheus Alerts#

Running the Installation#

Verification#

What This Does Not Cover#

Summary#