SSH Certificate Authority at Scale

Every organization running enough Linux hosts eventually has the same conversation:

• How do you authenticate SSH access for your employees?
• How do you authorize what they’re allowed to reach on the fleet?
• How do you manage those credentials centrally and consistently, without compromising availability?
• How do you revoke access, fast, when you need to?
• How do you make all of that audited and non-repudiable?

Problem

Imagine you’re on a team responsible for making SSH access self-service for 10K+ employees across 1M of hosts. Naive solutions get you pretty far at homelab or small-startup scale. They fall apart once you’re operating at hyperscaler scale.

This post covers what doesn’t work, what does, and — more usefully — the parts nobody mentions until they’re the reason you’re paged at 3am.

Disclaimer: none of this describes any particular employer’s deployment. It’s the reasoning behind the pattern, and the failure modes that are structural to it — not implementation details of a system I’ve worked on.

Approaches

There are many starting approaches, but most of them fail exactly when you can’t afford it.

Distributing SSH authorized_keys across hosts

This is standard OpenSSH configuration, and it’s usually where companies start. Every host keeps a file listing which SSH public keys may log in as each local user. The default location is /home/%u/.authorized_keys, where %u is the username token:

> cat /home/alice/.authorized_keys
ssh-rsa AAAAB[REDACTED]M3n alice@YK-234567

> cat /home/bob/.authorized_keys
ssh-rsa AAAAC[REDACTED]L4m bob@YK-123456

> cat /etc/ssh/sshd_config
...
AuthorizedKeysFile /home/%u/.ssh/authorized_keys
...

The model is easy to reason about and has no downtime from external dependencies, but it breaks down as soon as you have a non-trivial number of users or hosts. Keeping every host’s authorized_keys current becomes a hard distributed-systems problem in its own right.

Commonly, this becomes a job for a configuration system — SaltStack, Ansible, Chef — or something home-grown. That job usually takes one of two shapes:

Centralized sync job

One option: a centralized sync job that runs on a schedule — every couple of minutes, say — and pushes configuration to every target host over SSH or some other protocol. It has real limits:

Pros:

• Local-only authentication at the time of login
• Easy to deploy and control what it does

Cons:

• Scales linearly with host count, which gets expensive fast
• You’re guaranteed to breach your SLA unless you shard and scale the job out
• Propagation lag is in minutes
• Needs to reach every host directly, which NATs and private subnets make practically impossible
• Needs credentials to make sudo changes on every host — one more secret to manage

Background agent

A slightly better option: run an agent on every host instead. It calls home on a schedule and pulls the latest SSH data for that host.

Pros:

• Local-only authentication at the time of login
• Scales much better with the number of hosts
• NATs are no longer a problem — the agent on the host is the client, so it’s easy to make your data-plane servers reachable
• No need to maintain external log-in credentials — the agent itself runs with enough privilege to update SSH keys

Cons:

• If the agent breaks, you need break-glass access to reach and fix that host
• At 100K+ hosts, your data plane needs to withstand serious throughput — TLS handshakes included
• In a large org you’ll need to support multiple platforms to meet other teams where they are (Go is a good fit here, for what it’s worth)
• The agent has to stay out of the way of the production workloads running alongside it
• Propagation lag is in minutes
• It’s eventually consistent by design, with no hard guarantees
• A disconnected or misconfigured host can silently stop getting updates altogether

Using LDAP

Another option: point OpenSSH’s AuthorizedKeysCommand at LDAP or an internal directory, checked on every login.

But now every SSH login is a network call to LDAP, from every host, and the directory service becomes a hard dependency for getting SSH access anywhere. If it goes down — replication lag, a network partition, anything — so does everyone’s access.

That’s a real coupling, not a hypothetical one — the failure mode isn’t graceful degradation, it’s every host individually deciding it can’t authenticate anyone. Caching credentials for a while helps, but it does nothing for anyone who hasn’t logged into that host recently.

Pros:

• No need to keep hosts up-to-date after the initial setup
• Scales well with the number of hosts
• No more propagation lag — revoke in the directory, and the next login attempt sees it immediately

Cons:

• Hard dependency on the directory for any SSH access, unless cached
• At 100K+ hosts, LDAP clusters have to grow, and replication gets harder
• LDAP becomes a very high-value target for attackers

SSH Keys

Independent of how keys get onto a host, there’s still the question of what backs the keypair itself.

Soft SSH keys

It’s trivial to generate a keypair with ssh-keygen and publish the public half into a centralized registry — and just as trivial to leak the private half: an accidental commit, a rogue AI agent reading it off disk, malware that exfiltrates your ~/.ssh/ directory.

Often you won’t even know you’ve been compromised, unless you also have controls like a VPN perimeter, source-IP/location matching, or an incident response team watching for anomalous usage patterns.

Any compromise means immediately revoking that key everywhere, and confirming the attacker hasn’t left a backdoor on whatever hosts they reached.

This is rightly a non-starter for any company that’s serious about its business and its customers’ data.

Hardware Tokens

YubiKeys and their equivalents elegantly solve the credential-theft problem: the private key never leaves the device. The device only exposes its public key and can sign arbitrary strings, which is exactly what the SSH handshake needs.

There are far fewer ways for an attacker to get the the private key: a cryptographically relevant quantum computer (should one arrive), physically steal the hardware token, or mount a man-in-the-middle attack via agent forwarding.

Hardware tokens are great, but they trade the problem for a physical logistics one. Issuing them to employees worldwide, replacing lost or broken units, and deprovisioning when someone leaves the company all become supply-chain and shipping problems layered on top of your access control system. Cost scales roughly linearly with headcount, with no economies of scale to look forward to.

Each of these is a reasonable choice at some scale. None of them is a good choice at every scale, and the axis they all fail on is the same one: the credential and the access decision are the same artifact. The key that lets you in today is the key that lets you in tomorrow, unless someone does work to change that. Certificates decouple those two things.

SSH Certificate Authorities

An SSH certificate is a public key plus a signed statement about it: which principals may use it, which hosts it’s good for, when it expires, and what constraints apply while it’s in use.

> ssh-keygen -Lf ~/.ssh/id_rsa-cert.pub
    Type: ssh-rsa-cert-v01@openssh.com user certificate
    Public key: RSA-CERT SHA256:sXe[REDACTED]zAE
    Signing CA: RSA SHA256:14lk[REDACTED]MWw (using rsa-sha2-512)
    Key ID: "..."
    Serial: 12345
    Valid: from 2026-08-10T23:13:16Z to 2026-08-10T23:18:16Z
    Principals:
            bob@api-staging
    Critical Options: (none)
    Extensions:
            permit-port-forwarding
            permit-pty
    Signature: ...

All an OpenSSH server needs to do to allow access is:

• Verify the signature against the list of CAs in TrustedUserCAKeys
• Check the validity window
• Check that the certificate’s principals match the AuthorizedPrincipals of that user

That’s it — no network calls to LDAP, no background jobs or agents propagating credentials across millions of hosts. The certificate the client presents carries everything needed to make the authentication and authorization decision.

This inverts the propagation problem. Instead of pushing keys — or their revocation — to every host, you centrally control the issuing of certificates, and existing ones simply age out (passive revocation).

Revocation becomes a property of time rather than a property of successful propagation.

This authentication/authorization flow doesn’t actually care what kind of key gets embedded in the certificate. It could be a hardware token’s public key — but an even better approach is to generate a fresh ephemeral keypair locally each time, send a CSR (Certificate Signing Request) to the SSH-CA, and get back a certificate that’s short-lived and ephemeral end to end. Even if an attacker later recovers that private key, it’s already worthless.

All the user needs is a way to authenticate to the SSH-CA and request the principal scope they’re authorized for — commonly done with a longer-lived STS token issued by the cloud’s IAM service.

Where it’s not the right answer. For a fleet under a few hundred hosts with a stable operator population, the CA’s operational overhead — running the signing service, distributing the trusted CA key to every host, building the client tooling that requests and refreshes certificates automatically, on the fly — is real cost against a problem you may not have yet. Hardware token-based authorized_keys plus decent config management is a perfectly good decision at that scale, and rebuilding it three times as you grow linearly is often cheaper than building the general system on day one.

The CA earns its cost when:

• You have hundreds of teams, each with a different access-authorization scope
• Access changes often, e.g. just-in-time grants
• Each access event needs to be heavily audited

Certificate lifetime

The certificate’s time-to-live is the single parameter that does the most work in the whole system, and it’s a direct trade against three things at once.

Short lifetimes bound the blast radius of a stolen certificate — an attacker with a five-minute certificate has five minutes to attack the system, as no separate revocation action is required. But short lifetimes also mean more frequent reissuance, which means the issuance path has to be fast, available, and cheap enough to call constantly. And they interact badly with clock skew: a certificate valid from t0 to t0+300s is invalid the moment a host’s clock is more than a few seconds off from the CA’s, in either direction. At global fleet scale, some fraction of hosts always have clock drift, and short-lived certificates make that drift visible as authentication failures instead of silently ignorable. A common fix is to set validity to t0-60s through t0+240s, absorbing a bit of skew on either side.

Long lifetimes are gentler on issuance load and clock skew, but now you’re most of the way back to the standing-credential problem you built the CA to avoid — a certificate valid for a day is a bearer credential valid for a day, and if it’s compromised, you’re back to needing a real-time revocation mechanism after all.

Long-running sessions are the edge case that breaks whichever choice you make. An operator with an open SSH session doesn’t get disconnected when their certificate expires — the certificate was checked at connection time, not continuously. So a short TTL doesn’t actually bound an open session’s lifetime; it only bounds how long a new connection can be started with that certificate. If you need to cut off an active session, that’s a different mechanism (host-side session limits, or killing the sshd session directly), and it’s worth being honest that certificate TTL doesn’t solve this problem.

Scoping access via principals

By default, an SSH certificate’s principal is just the local username the holder is trying to log in as. That gives you no control over which host they use it against — the right one, or one that has nothing to do with their role. That’s where scoping comes in.

Principals are the list of “subjects” a certificate is valid for — commonly the local usernames it can authenticate as, e.g. dev, dba, root, or an employee’s unique POSIX account name.

But if you have a dba account on your payments fleet and another dba account on your identity fleet, how do you grant access to one but not the other? SSH certificates have no built-in host-restriction mechanism, but there’s a simple trick that gets you there:

1. Bind the “subject” to the fleet, e.g. dev@payments or auditor@identity.
2. Adjust the host’s sshd config so AuthorizedPrincipalsCommand prints %u@<fleet> instead of the default %u.

In this scheme, Alice can only log in as dev to the payments fleet and as auditor to the identity fleet — nothing more.

Depending on your organization’s requirements, you still need to answer the question of how granular your scoping needs to be, before the issuance itself becomes the bottleneck. A certificate scoped to exactly one host for exactly one task is maximally safe and requires an issuance system that can mint that many distinct certificates without becoming the thing operators wait on. Coarser scoping is operationally cheaper and philosophically closer to the standing access you started with.

Critical options and extensions

SSH certificates carry critical options that constrain what a session can do, not just who can start one.

force-command pins the session to a specific command, regardless of what the client requests — useful for service accounts that should only ever run one script, and nearly useless for interactive operator access, where the entire point is running arbitrary commands.

source-address restricts which client IPs the certificate is valid from. This helps when operator access is expected to originate from known infrastructure — a bastion, a VPN egress range — and does nothing if that range is broad or an attacker sits inside it.

Disabling port forwarding, agent forwarding, and PTY allocation each close a specific lateral-movement path. Agent forwarding is the one worth understanding even if you allow it: a forwarded agent lets the remote host ask the operator’s local agent to sign things, which means a compromised remote host can pivot through the operator’s credential to anywhere else it’s trusted, without ever seeing the private key itself. That’s a real primitive attackers use, and it’s off by default for a reason.

None of these options are free, in the sense that every one of them is a policy decision somebody has to own and be able to explain when it makes an operator’s workflow harder. A constraint nobody can justify gets routed around.

The CA key itself

The CA’s signing key is the single point that, if compromised, lets an attacker mint valid access to everything the CA is trusted for. This deserves treatment proportional to that fact, not proportional to how often it comes up in day-to-day operation.

Where it lives matters: an HSM or equivalent hardware boundary means the key material never exists in a form that can be exfiltrated wholesale, only used through a controlled signing interface. Quorum on its use — requiring more than one authorization to invoke certain operations, especially anything that touches the key itself rather than routine signing — turns “one compromised credential” into “a conspiracy,” which is a meaningfully higher bar.

The recovery story is the part almost nobody wants to think through before they need it: if the CA key is compromised, every certificate it ever issued is suspect, and every host that trusts it needs to start trusting a new key instead. That’s a fleet-wide trust rotation, and if you haven’t rehearsed it, the first time you do it will be during the incident that made it necessary. It’s worth treating “rotate the CA key without an outage” as a drill, not a hypothetical.

Revocation

Two real mechanisms exist, and they trade off against each other rather than one being strictly better.

Certificate Revocation Lists (CRLs), distributed to hosts, let you invalidate a specific certificate before it would naturally expire. This gives you real-time revocation independent of certificate lifetime — but it reintroduces the exact propagation problem the CA was supposed to solve: a CRL has to reach every host that might see the revoked certificate, and a host that hasn’t gotten the update will happily accept it.

Short certificate lifetimes sidestep propagation entirely — there’s nothing to distribute, because the certificate simply stops being valid. The cost is the issuance-load and clock-skew pressure discussed above.

In practice these aren’t either/or. Short lifetimes handle the common case — routine access naturally aging out — while KRLs handle the case short lifetimes can’t: revoking something right now, mid-lifetime, because you know something the expiration timer doesn’t. Treating KRLs as the break-glass revocation mechanism, and lifetime as the routine one, uses each for what it’s actually good at.

Host certificates

Most of this post is about proving who the operator is and what they can do. The other half of SSH trust — proving which host you’re connecting to — gets skipped far more often, and it’s where a very common, very human failure mode lives.

Without host certificates, SSH relies on trust-on-first-connect: the client remembers a host’s key fingerprint the first time it connects, and complains loudly if that fingerprint ever changes (TOFU). This is a reasonable model until host keys legitimately rotate — a bastion gets rebuilt, a fleet gets re-imaged — at which point every operator who’s ever connected gets a scary, correct-looking warning that a security control is supposed to produce when something bad happens, except this time nothing bad happened.

What operators do with that warning, reliably, is learn to bypass it — a flag that skips host verification, typed reflexively because it’s fired a dozen times for benign reasons before. That habit doesn’t stay scoped to the day the bastion rotated; it becomes muscle memory, invoked the next time too, including the time it isn’t benign. At that point the control hasn’t just failed — it’s actively trained the population it was protecting to disable it.

Host certificates fix this the same way client certificates fixed operator identity: a host presents a certificate signed by a CA the client already trusts, rotation becomes a non-event because the new key is still signed by the same trusted authority, and there’s never a legitimate reason to see the raw fingerprint-changed warning. If you’ve built the client-side half of this pattern and not the host-side half, this is usually the highest-value thing left on the table — it’s a small build, and it directly kills a workaround your own operators have probably already learned by heart.

Lessons Learned: What breaks in production

None of this shows up in a design doc. It shows up eight months after launch, and this is the section that would have saved everyone time if it had existed on day one.

Clock skew across a global fleet

Certificate validity is a time window, and a time window is only as meaningful as clock agreement between the CA and every host checking it. NTP is table stakes; it is not a guarantee. Hosts recovering from a hard reboot, running in restricted network environments, or living somewhere NTP silently stopped working will drift, and drift shows up as authentication failures that look exactly like a certificate bug. The fix isn’t a smarter certificate format — it’s monitoring clock drift as a first-class signal, before it manifests as a confusing login failure someone else has to debug from scratch.

The bootstrap problem

A client requesting a certificate has to authenticate to the CA somehow, and that authentication can’t itself be a certificate from the same CA — otherwise you’ve just moved the original problem one level up without solving it. That initial trust has to come from somewhere else entirely — platform-level identity, like the STS token from IAM mentioned above. Whatever it is, it’s now a second system with its own security properties, and it’s easy to spend all your design effort on the elegant certificate layer and comparatively little on the initial-trust mechanism underneath it — which is exactly where an attacker who can’t forge a certificate will go looking instead.

CA unavailability during an incident

The CA is a hard dependency for getting new access, which means the day it’s down is also, with unpleasant frequency, the day someone needs new access most urgently — because the CA usually shares failure domains with the rest of your infrastructure. This is the strongest argument for a break-glass path that doesn’t depend on the CA’s own availability: something orthogonal, rarely used, and tested on a schedule rather than trusted to work the first time it’s needed for real.

Client-side retry limits

OpenSSH’s default configuration (MaxAuthTries) rejects a client after 6 failed attempts. Connect to enough different hosts — different fleets, regions, or accounts — and you’ll hit Authentication Denied even though you got a valid certificate from SSH-CA every single time.

That means your client tooling — the thing that requests certificates on your employees’ behalf — has to actively manage what’s loaded:

• unload expired certificates automatically
• push the most recently minted certificate to the front of the SSH agent
• make sure agent key-management operations are safe under concurrent access

Migrating off standing keys

This is the part that’s organizationally hard rather than technically hard, and it’s consistently underestimated. Cutover means every existing workflow, script, and mental habit built around long-lived keys has to move to short-lived certificates before you can safely turn the old path off — and “before you can safely turn it off” is doing a lot of work in that sentence, because the old and new paths necessarily coexist for a while. That transition period, where both exist, is the highest-risk moment for the entire project: it’s when the standing keys you’re trying to retire are still valid, so you haven’t gained the security benefit yet, while you’ve already taken on the operational complexity of running two systems at once. Plan for that window to be longer than feels reasonable (years?), and be honest that the actual work is migrating people and their habits, not writing the CA.

Client-side tooling quality is non-negotiable

There are a lot of moving parts involved in getting a certificate with the right principals, and none of that should be the employee’s problem — they’re busy fixing an incident at 3am, not remembering which flags get them SSH access to a host.

When they type ssh oncall@{region}.{ad}.payments-3, the tooling needs to transparently:

• refresh their STS token if it’s expired or invalid
• resolve the principal needed for that host (oncall@payments, via SSH’s ProxyCommand/Match exec)
• fetch a certificate right before the SSH handshake
• load it into the SSH agent ahead of any others, safely under concurrent access
• fail with error messages good enough to act on

Unless all of that holds, teams won’t accept the extra overhead — they’ll go to your management asking to go back to the “good old days” of plain SSH keys.

Epilogue

This is probably my favorite project I’ve ever worked on, as it embodies what I love the most about the platform engineering:

You know you’ve done your job well when your infrastructure gets completely out of the way and your customers never have to think about all this Rube Goldberg machinery running under the hood just so they can log in to their machine.