Atomic Certificate Rotation

The Rotation Problem

TLS certificates expire. When iddio’s proxy cert expires, all agent connections fail — the TLS handshake rejects an expired certificate. Manual rotation (generate new cert, replace file, restart proxy) creates a maintenance window and a potential for human error.

Automatic certificate rotation eliminates this. The proxy monitors its own certificate’s expiry date and renews it before it expires, without downtime.

Rotation Architecture

A background goroutine checks the certificate’s expiry once per hour. When the certificate enters the renewal window (30 days before expiry by default), the proxy generates a new certificate, writes it atomically, and swaps it into the running TLS config:

func (p *Proxy) certRotationLoop(ctx context.Context) {
    ticker := time.NewTicker(1 * time.Hour)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            p.checkAndRotate()
        }
    }
}

func (p *Proxy) checkAndRotate() {
    cert, _ := tls.LoadX509KeyPair(p.certPath, p.keyPath)
    leaf, _ := x509.ParseCertificate(cert.Certificate[0])

    renewAfter := leaf.NotAfter.Add(-30 * 24 * time.Hour)
    if time.Now().Before(renewAfter) {
        return // Not yet in renewal window
    }

    log.Printf("certificate expires %s, renewing", leaf.NotAfter)
    p.rotateCert()
}

Atomic File Writes

Certificate files are written using the temp-and-rename pattern:

func atomicWriteFile(path string, data []byte, perm os.FileMode) error {
    dir := filepath.Dir(path)
    tmp, err := os.CreateTemp(dir, ".tmp-*")
    if err != nil {
        return err
    }

    if _, err := tmp.Write(data); err != nil {
        tmp.Close()
        os.Remove(tmp.Name())
        return err
    }

    if err := tmp.Chmod(perm); err != nil {
        tmp.Close()
        os.Remove(tmp.Name())
        return err
    }

    tmp.Close()

    // Rename is atomic on POSIX filesystems
    return os.Rename(tmp.Name(), path)
}

This ensures that at no point is the certificate file partially written. Either the old cert exists or the new cert exists — never a corrupted intermediate state. The os.Rename call is atomic on all POSIX filesystems.

SHA-256 Fingerprinting

After writing the new certificate, the proxy verifies it by computing its SHA-256 fingerprint and comparing it to the expected value:

func (p *Proxy) rotateCert() error {
    // Generate new cert signed by CA
    newCert, newKey, err := p.ca.IssueCert(p.hostname, 365*24*time.Hour)
    if err != nil {
        return fmt.Errorf("cert issue: %w", err)
    }

    // Write atomically
    if err := atomicWriteFile(p.certPath, newCert, 0600); err != nil {
        return err
    }
    if err := atomicWriteFile(p.keyPath, newKey, 0600); err != nil {
        return err
    }

    // Verify by loading and fingerprinting
    loaded, err := tls.LoadX509KeyPair(p.certPath, p.keyPath)
    if err != nil {
        return fmt.Errorf("cert verify: %w", err)
    }

    leaf, _ := x509.ParseCertificate(loaded.Certificate[0])
    fp := sha256.Sum256(leaf.Raw)
    log.Printf("certificate rotated: fingerprint=%s, expires=%s",
        hex.EncodeToString(fp[:8]), leaf.NotAfter)

    // Swap into TLS config
    p.tlsConfig.Certificates = []tls.Certificate{loaded}
    return nil
}

The fingerprint is logged for audit purposes. If you need to verify that a specific certificate is in use, compare the logged fingerprint against the certificate file.

TLS Config Swap

Go’s tls.Config supports dynamic certificate selection via GetCertificate. Iddio uses this to swap certificates without restarting the listener:

tlsConfig := &tls.Config{
    GetCertificate: func(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
        p.certMu.RLock()
        defer p.certMu.RUnlock()
        return p.currentCert, nil
    },
    ClientAuth: tls.RequireAndVerifyClientCert,
    ClientCAs:  p.caPool,
}

When the certificate is rotated, the proxy acquires a write lock, swaps the cert reference, and releases the lock. In-flight TLS handshakes using the old cert complete normally. New handshakes use the new cert.

Rotation Timeline

Event	Timing
Certificate issued	Day 0
Check interval	Every 1 hour
Renewal window opens	Day 335 (30 days before expiry)
New cert generated	First check after window opens
Certificate expires	Day 365

With a 365-day certificate and a 30-day renewal window, you have a full month of buffer. Even if the renewal fails repeatedly, the proxy will retry every hour — 720 attempts before the old cert expires.

Agent Certificates

Agent certificates are also rotated, but through a different mechanism. Since agents connect to the proxy (not the other way around), agent cert rotation is triggered by the iddio agent renew command or automatically by the enterprise control plane.

Agent certs use the same atomic write + fingerprint verification pattern. The agent doesn’t need to restart — kubectl re-reads the client cert on each connection.