SSH certificate authority at scale

Imagine you’re on a team responsible for making SSH access self-service for more than 10,000 employees across one million hosts. You need to authenticate every operator, authorize what they can reach, withdraw access quickly, and audit the result without making the access system a new production failure mode.

Simple solutions work well for a homelab or a small startup. At hyperscaler scale, their common problem is the amount of durable, per-user authorization state that must stay synchronized across the fleet. SSH certificates replace those per-key trust entries with a small set of certificate authority (CA) trust anchors. The issuer makes a time-bounded access decision, the certificate carries that decision, and each host applies its local policy without contacting the issuer.

This post explains which approaches stop scaling, what certificates change, and which operational problems appear only after the design reaches production.

Disclaimer: None of this describes a particular employer’s deployment. It covers the reasoning behind the pattern and the failure modes inherent to it, not the implementation details of a system I’ve worked on.

Why per-user host state stops scaling

Distributing `authorized_keys`

This is standard OpenSSH configuration, and it’s usually where companies start. Each local account has a file listing the public keys that may authenticate as that user. A relative AuthorizedKeysFile path is resolved from the account’s home directory; ~/.ssh/authorized_keys is the familiar location, but the home directory doesn’t have to be under /home.

The following example shows two accounts with separate authorized-key files:

$ cat /home/alice/.ssh/authorized_keys
ssh-rsa AAAAB[REDACTED]M3n alice@20250809

$ cat /home/bob/.ssh/authorized_keys
ssh-rsa AAAAC[REDACTED]L4m bob@20260903

The model is easy to reason about and has no external dependency during sign-in. Keeping every host’s copy current becomes a distributed-systems problem as the number of users and hosts grows. That work usually lands on a configuration system such as Salt, Ansible, Chef, or an internal service.

A centralized push service performs work proportional to the number of hosts. It also needs a route and a privileged update mechanism to each one. Network address translation (NAT), private subnets, and intermittently connected machines require relays or additional control channels, and revocation speed depends on how quickly the service reaches the slowest relevant host.

A pull agent avoids inbound reachability. It trades that problem for polling and partition lag: a disconnected or misconfigured host can continue using stale authorization state. The agent also shares resources and a failure domain with the production workload. If it breaks, repairing the host may require the same access path the agent was meant to maintain.

Checking a directory during authentication

Another option is to configure AuthorizedKeysCommand to run a local helper that queries Lightweight Directory Access Protocol (LDAP) or an internal directory for each authentication attempt. A direct lookup removes host-side key-propagation lag: the next lookup can observe updated state as soon as the directory makes it available.

The cost is an online dependency in the authentication path. If the helper cannot reach the directory, hosts must either reject access or rely on cached results. Caching improves availability, but its scope and expiration policy determine how quickly revocation takes effect. The directory and its credentials also become high-value security targets.

What SSH certificates change

An OpenSSH certificate is a public key plus signed identity information: its type, serial number, key ID, principals, validity interval, critical options, and extensions. OpenSSH defines separate user and host certificates. User certificates authenticate people or workloads to servers; host certificates authenticate servers to clients. They are not X.509 certificates.

The following output shows a five-minute user certificate:

$ ssh-keygen -Lf ~/.ssh/id_rsa-cert.pub
    Type: ssh-rsa-cert-v01@openssh.com user certificate
    Public key: RSA-CERT SHA256:sXe[REDACTED]zAE
    Signing CA: RSA SHA256:14lk[REDACTED]MWw (using rsa-sha2-512)
    Key ID: "..."
    Serial: 12345
    Valid: from 2026-08-10T23:13:16Z to 2026-08-10T23:18:16Z
    Principals:
            bob@api-staging
    Critical Options: (none)
    Extensions:
            permit-port-forwarding
            permit-pty
    Signature: ...

A typical authentication flow has the following sequence:

The client generates an ephemeral key pair or selects a hardware-backed key.
The operator authenticates to the issuer through an independent identity system and requests an authorized principal scope.
The client sends the public key and requested scope to the issuer.
The issuer evaluates policy and signs the public key with the approved principals, options, and validity interval.
The client presents the certificate and proves possession of the corresponding private key during the SSH handshake.
The server evaluates the certificate and its local policy.

For user-certificate authentication, an OpenSSH server checks at least the following conditions:

The certificate has the correct type and a signature from an allowed user CA.
The signature algorithm is allowed, and the certificate is within its validity interval.
Neither the subject key nor the certificate is listed in the configured revoked keys data.
The client proves possession of the private key that matches the certified public key.
A certificate principal matches the target account or the host’s configured authorized-principal policy.
The certificate’s critical options are understood and satisfied, its permitted extensions are applied, and normal account and server policy allow the session.

If principal resolution and account policy are local, the host makes this decision without a network call to the issuer or a directory. Hosts still need the trusted CA keys, authorization configuration, and any Key Revocation List (KRL). Certificates don’t eliminate distribution; they replace frequently changing per-user keys with smaller, slower-changing trust and policy state.

Protecting the subject key

Certificates don’t remove the private key from authentication. They change how long and where the matching public key is trusted.

Software-backed keys are easy to copy if malware or another process can read them from disk or memory. Passphrases, encrypted storage, and carefully constrained SSH agents reduce the risk, but a copied private key remains usable anywhere its public key or a matching, still-valid certificate is trusted.

Hardware-backed keys reduce extractability. For example, OpenSSH FIDO keys combine a key handle, which may be stored on disk, with per-device private material that the authenticator doesn’t export. A compromised workstation or forwarded agent can still ask the device to sign while it is available, and a stolen device may be usable if its presence or verification requirements can be satisfied.

Another pattern is to generate a fresh key pair for each issuance, send only its public key in the signing request, and discard the private key after the certificate expires. If that public key isn’t trusted anywhere else, recovering the private key after expiration provides no SSH access.

The CA becomes worthwhile when the cost of managing it is lower than the cost and risk of distributing individual keys. Common signals include:

Many teams require different authorization scopes.
Access changes frequently, including just-in-time grants.
Each access decision requires a durable audit trail.
Per-user key updates cannot meet the fleet’s propagation objective.

For a fleet of a few hundred hosts with a stable operator population, the CA’s operational overhead may cost more than the problem it solves. Hardware-backed keys in authorized_keys, combined with reliable configuration management, can remain a good choice at that scale.

Encoding authorization

Principal scope

The CA chooses the principal list when it signs a user certificate. If neither AuthorizedPrincipalsFile nor AuthorizedPrincipalsCommand is configured, sshd requires the target local username to appear in that list. This default identifies which account the holder may use, but it doesn’t restrict which host accepts the same account name.

Fleet-qualified principals add that scope through host policy. For example:

The issuer includes dev@payments or auditor@identity instead of the generic principals dev and auditor.
Each host accepts only the principal formed from the target username and its fleet identity. A payments host evaluating a sign-in as dev accepts dev@payments, not dev@identity.

AuthorizedPrincipalsCommand can generate that list from local, trusted metadata. OpenSSH requires the command to run as the account configured by AuthorizedPrincipalsCommandUser; a dedicated unprivileged account is the safer choice. AuthorizedPrincipalsFile provides the same matching behavior through static files. If the command queries a central service during every sign-in, the design reintroduces the online dependency that certificates otherwise avoid.

This scheme doesn’t add a host field to the user certificate. It works because each fleet accepts a different namespace of principals. A certificate containing only dev@payments therefore works only on hosts whose local policy accepts that value.

Finer scopes reduce what one certificate can reach, but they increase issuance volume and policy complexity. A one-host, one-task certificate limits exposure more than a fleet-wide certificate, but only if the issuer can mint it without delaying operators or creating a less reliable access path.

Critical options and extensions

OpenSSH treats critical options and extensions differently. A server rejects a certificate containing a critical option it doesn’t understand. It may ignore an unknown extension. Standard user-certificate extensions grant capabilities such as port forwarding, agent forwarding, pseudo-terminal (PTY) allocation, X11 forwarding, and execution of ~/.ssh/rc.

force-command replaces the shell or command requested by the client with a fixed command. It is useful for service accounts that perform one operation and usually incompatible with general interactive access.

source-address restricts the source addresses from which sshd considers the certificate valid. The restriction applies to the address observed by the server, which may be a bastion or proxy address rather than the operator’s workstation.

Certificate permissions are one layer of session policy. For agent forwarding, the certificate must permit it, the server must allow it, and the client must request it. The OpenSSH client disables ForwardAgent by default, while certificates made with ssh-keygen permit agent forwarding unless the signer clears or disables that permission.

Agent forwarding deserves particular care. A compromised remote host cannot export private key material from the forwarded agent, but it can ask the agent to perform authentication operations with loaded identities. Destination-constrained agent keys and jump hosts reduce that exposure.

Omitting port-forwarding and PTY permissions can still be useful, especially for a forced-command or otherwise restricted account. They aren’t a security boundary for an unrestricted shell: a user who can run arbitrary programs can create another forwarder, and disabling PTY allocation doesn’t disable command execution. Apply each restriction as part of a complete account policy, not as an isolated flag.

Lifetime and revocation

Certificate lifetime and clock skew

Certificate lifetime directly trades exposure against availability and issuance cost. A short-lived certificate limits how long someone holding both the certificate and its private key can start new sessions. After it expires, routine cleanup needs no revocation update. Short lifetimes also require a fast, available, and inexpensive issuance path because clients call it frequently.

Clock skew matters near both validity boundaries. If a certificate becomes valid at t0, a host whose clock is behind may reject it immediately after issuance. A host whose clock is ahead may reject it before its intended expiration. Backdating valid_after by the tolerated skew reduces failures at issuance. The valid_before value must then account for the desired usable lifetime and the additional exposure created by that tolerance.

For example, t0-60s through t0+300s creates a six-minute encoded interval and five future minutes for a synchronized host. A fast host expires it sooner, while a slow host accepts it longer in real time. There is no free skew allowance: every extra second in the interval increases the maximum credential lifetime.

Longer lifetimes reduce issuance load and make clock error less visible. They also extend the period in which a compromised certificate and private key can open a new session. At that point, emergency revocation becomes more important.

Certificate expiration doesn’t end an established SSH connection. sshd checks the certificate during authentication, not continuously throughout the session. A short lifetime therefore bounds new connections, not existing ones. Ending an active session requires a separate control, such as a session limit or terminating the server-side process.

Pre-expiry revocation

OpenSSH KRLs can revoke public keys or certificates. A KRL can select certificates by serial number or key ID without requiring the original certificate. sshd consults the file configured by RevokedKeys during public-key authentication.

A KRL provides pre-expiry revocation only after the updated file reaches a host. It therefore reintroduces a propagation problem: a host with stale revocation data may accept a certificate that another host rejects. Updates should replace the file atomically because an unreadable RevokedKeys file causes sshd to reject all public-key authentication.

In practice, lifetime and KRLs serve different cases. Short lifetimes bound routine exposure without distributing per-certificate updates. KRLs handle the exceptional case that cannot wait for expiration. Treating the KRL as an emergency control keeps its propagation burden out of the common path without pretending that expiration is immediate revocation.

Operating the issuer

The CA key

Compromise of a CA signing key lets an attacker mint certificates for every scope that trusts that CA. Protect the key according to that blast radius.

A hardware security module (HSM) or equivalent boundary helps when the signing key is generated as non-exportable and remains inside it. Backups must receive equivalent protection. The service invokes signing through a controlled interface instead of loading raw key material into application memory.

Multi-party authorization for sensitive key-management operations reduces the risk that one compromised administrator credential can replace, misuse, or destroy the key. Routine signing should remain automated and policy-bound; human approval on every certificate would turn the CA into an availability bottleneck.

After a suspected CA-key compromise, an attacker may have issued certificates that are indistinguishable from legitimate ones. Hosts must begin trusting a replacement key and then stop trusting the old one, which invalidates every still-valid certificate under that CA. Rehearse this fleet-wide trust rotation before an incident requires it.

Bootstrap identity

A client must authenticate to the issuer before it receives a certificate. A still-valid certificate from the same CA can support renewal, but it cannot be the only root of trust: initial enrollment, lost credentials, policy changes, and recovery need an independent identity source.

That source might be a workforce identity provider, a platform workload identity, or a temporary token from a cloud Security Token Service (STS) governed by Identity and Access Management (IAM) policy. Its assurance, revocation, and availability are part of the SSH system’s security model. An attacker who cannot forge a certificate will look for a weaker path through the bootstrap identity or issuer policy.

Issuer availability and break-glass access

The issuer is a hard dependency for obtaining new certificates, but existing unexpired certificates continue to work while it is unavailable. Shorter lifetimes make an issuer outage visible sooner.

Run the issuer across failure domains that don’t collapse with the infrastructure it protects. Maintain a break-glass path that is independent of the issuer and its primary identity provider. Keep that path narrowly scoped, monitor every use, and test it on a schedule rather than trusting it for the first time during an incident.

Audit trail

Audit the decision at issuance and the use of the certificate at each host. An issuance record should capture the authenticated subject, policy inputs, requested and granted principals, options, public-key fingerprint, certificate serial and key ID, validity interval, decision, and request identifier.

Host authentication logs should connect the certificate serial and key ID to the target account, destination host, source address, CA, result, and server-side request or session identifier. Stable serials and key IDs make it possible to correlate the issuer’s decision with the host’s observation.

Authentication logs do not provide command-level accountability. If the threat model requires recording interactive commands or terminal output, add operating system auditing or a session-recording layer and protect those records from the accounts being observed. Synchronize clocks and define retention before an incident; otherwise, technically complete logs may still be impossible to correlate.

Making the client dependable

Host certificates

User certificates prove who the operator is. Host certificates prove which server the operator reached.

When host keys aren’t provisioned centrally, SSH commonly uses trust on first use (TOFU): the client records a host key during the first connection and warns if that key later changes. Global known_hosts files, KnownHostsCommand, DNS SSHFP records, and UpdateHostKeys provide other ways to establish or rotate host trust.

At fleet scale, legitimate host-key changes may be frequent enough that operators become accustomed to warnings. A rebuilt bastion or re-imaged host produces the same warning that protects against a machine-in-the-middle attack. Repeated benign warnings encourage unsafe workarounds such as disabling host verification.

With host certificates, the client trusts a host CA through an @cert-authority entry in a known-hosts file. It accepts a host certificate only when the CA matches that trust entry and a certificate principal matches the requested hostname. A host can rotate its key without a changed-key prompt as long as the replacement certificate remains valid under the same trust and naming policy.

Host certificates don’t remove every source of host-verification errors. Expired certificates, incorrect principals, CA rotation, and conflicting known-hosts entries still fail safely. They do remove the need to distribute every individual host key to every client.

Selecting the right identity

OpenSSH’s MaxAuthTries defaults to six attempts per connection. Unless IdentitiesOnly is enabled, the client may also try identities offered by its SSH agent. A client with many unrelated identities can exhaust the server’s attempt limit before reaching the correct key and certificate, producing Too many authentication failures despite having valid credentials.

Client tooling should select the identity explicitly instead of depending on the ordering of a shared agent. OpenSSH provides several controls:

IdentitiesOnly yes limits authentication to configured identity and certificate files.
IdentityFile selects the private key, or a public key whose private half is available from an agent.
CertificateFile selects the matching user certificate.
IdentityAgent selects a dedicated agent socket, isolating credentials for a particular tool or environment.

Time-limited agent entries can clean up expired credentials automatically. A dedicated agent or explicit files also avoid races between terminals that add, remove, or reorder identities in one shared agent.

Client-side workflow

An operator responding to an incident shouldn’t have to remember the issuer’s API or determine which principal matches a host. A command such as the following should be enough:

ssh oncall@api-3.us-west.example.net

Before the SSH handshake, client tooling should perform the following work:

Refresh the independent identity token if it has expired.
Derive the allowed principal scope from the canonical target.
Generate or select the private key for this issuance.
Request a certificate with the approved scope.
Launch ssh with the matching identity and certificate selected explicitly.
Report failures with enough context for the operator to act.

This orchestration usually belongs in a wrapper or credential helper. ProxyCommand controls the byte transport to the server; it has no principal semantics. Match exec can conditionally select SSH configuration based on a command’s exit status, but it is not itself an identity or principal resolver.

Migrating from standing keys

Migration is organizationally harder than building the signer. Every workflow, script, and habit built around long-lived keys must move before the old path can be disabled. During the transition, static keys and certificates coexist, so the fleet has the operational complexity of both systems without the full security benefit of retiring the old one.

Inventory human and machine access separately. Move interactive operators first if the client workflow is mature, then migrate automation according to its renewal and availability requirements. Measure use of the old path, assign owners to remaining keys, and remove each fallback only after its traffic reaches zero and its recovery path has been tested.

For a large organization, the overlap may last quarters or years. Plan for that period as a supported operating mode rather than treating it as a short deployment phase. The difficult work is changing dependencies and behavior, not signing the first certificate.

Epilogue

This is probably my favorite project I’ve worked on because it embodies what I love most about platform engineering:

You know you’ve done your job well when the infrastructure gets out of the way and operators never have to think about the machinery underneath it just to reach a machine.