OpenSSH has a ton of underutilized features and one in particular comes up a lot in my conversations, the certificate functionality. I have interacted with a ton of production systems in cloud and non-cloud environments that tend to glaze over SSH known_hosts trust on first use (TOFU) model and instead do crazy things like periodic copying of ssh key configs around, synchronized known_hosts to developers, or copy over the same private key material between hosts (yes this really does happen, please stop).
TOFU models work great for small shops and deployments or with a few people to maintain things, but once growth hits I find that people often start relying on things like containerization or immutable infrastructure cloud deployments where they invest heavily on Ansible or other forms of configuration management. The moment this starts happening TOFU starts to fall apart and developers without good guidance start to do what they do best, build stuff. The problem is that this often means relying on -o StrictHostKeyChecking=no. This opens up the entire management systems to proper Man-in-the-Middle attacks and key substitution attacks, and even worse socializes people to click past warning messages.
OpenSSH introduced the ability to use a certificate authority in OpenSSH 5.4, which has a similar structure to X.509 certificates, but much more simplified. Fundamentally this allows for organizations to generate a certificate authority set of SSH keys, sign public keys for users and hosts, and no longer rely on the TOFU model. When this mode is configured the host no longer has to be aware of individual keys, but instead the certificate authority, the key identity, and a set of approved “principals” that have access to a host. These principals are essentially a set of simple string names that are added by the Certificate Authority (CA) signing phase that give SSH the ability to contextualize about the approved keys. This can be as simple as setting the username or hostname of the signing request, or allow for specific role based groups to be applied. These principals can directly be referenced in the sshd_config(5) by matching and explicit rules can be applied for restrictions, but can additionally be added to the keys themselves! This allows for not only the host to apply rules to the keys, but the signing authority. This is absolutely ideal for automated use keys and even could be used to apply restrictions to short lived keys.
This post is the steps I use to set up and deploy OpenSSH PKI’s for my personal environment, nothing stands in place of the man pages.
In order to establish a OpenSSH PKI the following rough steps will be taken:
I have decided to use roughly the following directory structure for my CA:
.
├── ca
│ ├── host
│ │ ├── incoming
│ │ │ └── ca.hosakacorp.net.pub
│ │ └── outgoing
│ │ └── ca.hosakacorp.net-cert.pub
│ ├── serial
│ ├── ssh_ca_ed25519
│ ├── ssh_ca_ed25519.pub
│ ├── ssh_revoked_keys
│ └── user
│ ├── incoming
│ │ ├── _deploy
│ │ │ └── ssh_ed25519.pub
│ │ └── poptart
│ │ └── ssh_ed25519.pub
│ └── outgoing
│ ├── _deploy
│ │ └── ssh_ed25519-cert.pub
│ └── poptart
│ └── ssh_ed25519-cert.pub
├── principals
│ ├── _deploy
│ └── poptart
├── ssh_config
├── ssh_known_hosts
├── ssh_host_ed25519_key
├── ssh_host_ed25519_key-cert.pub
├── ssh_host_ed25519_key.pub
└── sshd_config
The very first thing that needs to be done is to generate the certificate authority keys. These are the most important keys you own. Even more than with TLS keys these should be protected and restricted to as few people as possible and as proper protections should be put in place (OpenSSH even has support for PKCS#11 tokens). The next step is to actually generate the keys, as a note I will be generating keys exclusively using the ed25519 algorithm and will be configuring the system to use this algorithm exclusively.
Key generation is, for the most part, exactly the same as normal SSH keys:
# mkdir -pm 755 '/etc/ssh/ca/'
# ssh-keygen -t ed25519 -f /etc/ssh/ca/ssh_ca_ed25519
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /etc/ssh/ca/ssh_ca_ed25519.
Your public key has been saved in /etc/ssh/ca/ssh_ca_ed25519.pub.
The key fingerprint is:
SHA256:a2qKtUqfuIskFdHSS+7RK8eX2p73F9kYp419+aQ2NBA root@ca.hosakacorp.net
The key's randomart image is:
+--[ED25519 256]--+
| .o |
| .. |
| .E E |
| .+ . . o|
| .. o .S. o B.|
| . o + o. . B o|
|... .o +o . B |
|oo = .=..|
|. *+=o..o. .ooo.o|
+----[SHA256]-----+
In order for hosts to use the PKI we need to create a format and process for the CA to handle signing requests and then create the signing request for the host. I have decided to build my PKI in the /etc/ssh/ca directory with a set of subdirectories for hosts and users, each of which contain directories for incoming signing requests and successfully signed keys (incoming and outgoing respectively). While this isn’t strictly required it does help with automated processes and backups. Additionally, the following code will create a signing request for the CA host itself in order to give you an idea of what that looks like.
mkdir -pm 755 "/etc/ssh/ca/host/incoming" "/etc/ssh/ca/host/outgoing" "/etc/ssh/ca/user/incoming" "/etc/ssh/ca/user/outgoing"
cp "/etc/ssh/ssh_host_ed25519_key.pub" "/etc/ssh/ca/host/incoming/ca.hosakacorp.net.pub"
# -I indicates the key identity
# -n indicates principals, this will be covered later, but these are just strings that can be applied to the keys
# -V is the validity, I decided to make this key valid for this host for 10 years from now, in reality I use a much shorter time and have a key rotation process
# -z is the serial number for revocation and identity reference
# -f should work but there appears to be a bug that prevents the output changing, will need to be reported upstream
ssh-keygen -s "/etc/ssh/ca/ssh_ca_ed25519" -h -I "ca.hosakacorp.net" -n "ca.hosakacorp.net,server" -V "-1m:+520w" -z 1 "/etc/ssh/ca/host/incoming/ca.hosakacorp.net.pub"
mv "/etc/ssh/ca/host/incoming/ca.hosakacorp.net-cert.pub" "/etc/ssh/ca/host/outgoing/ca.hosakacorp.net-cert.pub"
# copy newly signed key to the traditional ssh public key location with a `-cert` added
cp "/etc/ssh/ca/host/outgoing/ca.hosakacorp.net-cert.pub" "/etc/ssh/ssh_host_ed25519_key-cert.pub"
printf "1\\n" > "/etc/ssh/ca/serial"
The key itself can be analyzed using ssh-keygen(1), and the information validated:
ssh-keygen -L -f "/etc/ssh/ssh_host_ed25519_key-cert.pub"
/etc/ssh/ssh_host_ed25519_key-cert.pub:
Type: ssh-ed25519-cert-v01@openssh.com host certificate
Public key: ED25519-CERT SHA256:pdjY7SJknGheBqZCafzr+LuTf/sfLeXmqYCywJ2RwQU
Signing CA: ED25519 SHA256:a2qKtUqfuIskFdHSS+7RK8eX2p73F9kYp419+aQ2NBA (using ssh-ed25519)
Key ID: "ca.hosakacorp.net"
Serial: 1
Valid: from 2020-02-18T19:31:57 to 2030-02-05T19:32:57
Principals:
ca.hosakacorp.net
server
Critical Options: (none)
Extensions: (none)
In order to actually make the PKI structure work we need to sign user keys, change a few lines of sshd_config(5), and set known_hosts to identify the PKI. The first step is to create those keys and get them signed by the PKI, it should be noted that just like with a X.509 PKI you just need the public key signed and do not have to pass around private key material:
mkdir -pm 755 "/etc/ssh/ca/user/incoming/poptart/" "/etc/ssh/ca/user/outgoing/poptart/"
cp "$HOME/.ssh/id_ed25519.pub" "/etc/ssh/ca/user/incoming/poptart/ssh_ed25519.pub"
# the -z reads in the serial number, uses bc to calculate the next one, and writes it to the serial file if the key signing was successful
ssh-keygen -s "/etc/ssh/ca/ssh_ca_ed25519" -I "/etc/ssh/ca/user/incoming/poptart/ssh_ed25519.pub" -z "$(bc -e "$(cat /etc/ssh/ca/serial)+1" -e quit)" -V "-1m:+260w" -f "/etc/ssh/ca/user/outgoing/poptart/ssh_ed25519-cert.pub" -n "poptart,users,ca-managers" "/etc/ssh/ca/user/incoming/poptart/ssh_ed25519.pub" && bc -e "$(cat /etc/ssh/ca/serial)+1" -e quit | tee /etc/ssh/ca/serial
# again remember that there is a bug with the output specification of -f
mv "/etc/ssh/ca/user/incoming/poptart/ssh_ed25519-cert.pub" "/etc/ssh/ca/user/outgoing/poptart/ssh_ed25519-cert.pub"
ssh-keygen -L -f "/etc/ssh/ca/user/outgoing/poptart/ssh_ed25519-cert.pub"
cp "/etc/ssh/ca/user/outgoing/poptart/ssh_ed25519-cert.pub" "$HOME/.ssh/id_ed5519-cert.pub"
OpenSSH is kind enough to automatically look up identity files that contain a trailing -cert.pub when using the default configuration, so just placing these in your .ssh directory will use them by default. Now that we have a set of user keys and the CA server host keys configured, we need to make a couple of modifications to the sshd_config(5).
AuthorizedPrincipalsFile /etc/ssh/authorized_principals/%u
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
TrustedUserCAKeys /etc/ssh/ca/ssh_ca_ed25519.pub
CASignatureAlgorithms ssh-ed25519
# this will be covered later
RevokedKeys /etc/ssh/ca/ssh_revoked_keys
Now that the sshd config is changed, it is necessary to create the principal files for our user, officially authorizing them to gain ssh access to the host. Notice how this simply tells the host which principals are approved by the TrustedUserCAKeys and doesn’t actually require any key materials. The following command authorizes the poptart user into the system when the ca-managers principal is signed by the CA:
mkdir -pm 755 "/etc/ssh/authorized_principals"
printf "%s\\\\n" "ca-managers" >> "/etc/ssh/authorized_principals/poptart"
After issuing a reload or restart to the sshd service the last thing that needs to be done is to set the system wide known_hosts file to contain the certificate authority, allowing for inherited trust between keys in the network and suppressing the key is not trusted warning. This can be done by using the special syntax @cert-authority, which informs the clients in the network that all principals will be excepted from the CA.
# cat /etc/ssh/ssh_known_hosts
@cert-authority *.hosakacorp.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIELloYQCFeo8whLWX5E2DWgntO1BCIehp7GFrkpbRRk9 ca@hosakacorp.net
@cert-authority hosakacorp.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIELloYQCFeo8whLWX5E2DWgntO1BCIehp7GFrkpbRRk9 ca@hosakacorp.net
Once the ssh_known_hosts file is configured to accept principals and identities from our CA we can actually log into our authorized server and see what it looks like in the logs. Below you can see a redacted list of what verbose logging will do on our servers, showing that my identity (poptart@hosakacorp.net) is authorized for the user poptart:
2020-02-06T20:24:55.647176+00:00 perhonen sshd[17204]: Connection from 10.0.1.7 port 18265 on 10.0.3.1 port 22 rdomain ""
2020-02-06T20:24:56.462177+00:00 perhonen sshd[17204]: Failed publickey for poptart from 10.0.1.7 port 18265 ssh2: ED25519 SHA256:Oz2vZgzxhyINBkG94gz5gk5aCIKiJaefP81NXwSTfI7
2020-02-06T20:24:56.610086+00:00 perhonen sshd[17204]: Accepted certificate ID "poptart@hosakacorp.net" (serial 2) signed by ED25519 CA SHA256:a2qKtUqfuIskFdHSS+7RK8eX2p73F9kYp419+aQ2NBA via /etc/ssh/ca/ssh_ca_ed25519.pub
2020-02-06T20:24:56.619851+00:00 perhonen sshd[17204]: Postponed publickey for poptart from 10.0.1.7 port 18265 ssh2 [preauth]
2020-02-06T20:24:56.761613+00:00 perhonen sshd[17204]: Accepted certificate ID "poptart@hosakacorp.net" (serial 2) signed by ED25519 CA SHA256:a2qKtUqfuIskFdHSS+7RK8eX2p73F9kYp419+aQ2NBA via /etc/ssh/ca/ssh_ca_ed25519.pub
2020-02-06T20:24:56.803296+00:00 perhonen sshd[17204]: Accepted publickey for poptart from 10.0.1.7 port 18265 ssh2: ED25519-CERT SHA256:Oz2vZgzxhyINBkG94gz5gk5aCIKiJaefP81NXwSTfI7 ID poptart@hosakacorp.net (serial 2) CA ED25519 SHA256:a2qKtUqfuIskFdHSS+7RK8eX2p73F9kYp419+aQ2NBA
OpenSSH logs the principals that were authorized for the users, so keeping track of things in the logs can be done easily by looking for the identity or any additionally approved principals.
Key revocation quickly becomes a sore spot of any PKI and just like with TLS certificates a revocation list of some sort needs to be distributed. Fundamentally, what do you do if you have to explicitly prevent a compromised set of keys from being trusted, after all they are signed and approved. OpenSSH ssh-keygen(1) directly has support for generating and maintaining KRL lists:
WARNING: This will revoke the serial id 2, so make sure you understand what is going on
# Generate the krl and add the -z 2 (aka the `poptart` user from before) serial.
ssh-keygen -k -f "/etc/ssh/ca/ssh_revoked_keys" -s "/etc/ssh/ca/ssh_ca_ed25519.pub" -z 2 "/etc/ssh/user/outgoing/poptart/ssh_ed25519-cert.pub"
Once the revocation list is created or updated, it needs be distributed and updated on each of the hosts. This can also be done automatically via cron or some other configuration management system, I personally using signify(1) to distribute from a webserver and then a simple cron job to update it. The following snippet shows how to revoke a key and how to update the revocation list to deny the host:
# update the current krl by revoking the serial number 2 key
ssh-keygen -k -f "/etc/ssh/ca/ssh_revoked_keys" -u -s "/etc/ssh/ca/ssh_ca_ed25519.pub" -z 2 "/etc/ssh/ca/user/outgoing/poptart/id_ed25519-cert.pub"
# check to see if the key is revoked
ssh-keygen -Q -f "/etc/ssh/ca/ssh_revoked_keys" "/etc/ssh/ca/user/outgoing/poptart/id_ed25519-cert.pub"
The next step is to distribute CA public keys (either dynamically or through gold images). I distribute these on essentially all internal “leaders” or trusted hosts, and embed them in core Ansible deployments. I have attached a few helpful scripts to the bottom of this post that give an idea of how I dynamically distribute my CA keys, but in reality when doing custom builds or gold images they are default built in.
The following example is a complete step for enrolling a new server into the PKI and then allowing the users principal to have access to the poptart user. The shell signature perhonen # is the server I am trying to enroll and ca # is the CA, both running as root.
perhonen # mkdir -pm 755 "/etc/ssh/ca" "/etc/ssh/authorized_principals"
perhonen # printf "%s\\\\n" "users" >> "/etc/ssh/authorized_principals/poptart"
perhonen # printf "%s\\\\n" "www-deploy" >> "/etc/ssh/authorized_principals/_deploy"
ca # scp /etc/ssh/ca/ssh_ca_ed25519.pub root@perhonen.hosakacorp.net:/etc/ssh/ca
ca # scp /etc/ssh/sshd_config root@perhonen.hosakacorp.net:/etc/ssh/sshd_config
ca # scp /etc/ssh/ssh_known_hosts root@perhonen.hosakacorp.net:/etc/ssh/sshd_config
As usual when messing with remote authentication you can accidentally lock yourself out pretty easily so keep a session open for debugging and reload the service. Then test the configuration, the following shows an example from one of my previous configurations:
$ ssh -i ~/.ssh/id_ed25519 perhonen.hosakacorp.net -v
OpenSSH_8.1, LibreSSL 3.0.2
debug1: Reading configuration data /home/poptart/.ssh/config
debug1: /home/poptart/.ssh/config line 48: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to perhonen [...] port 22.
debug1: Connection established.
debug1: identity file /home/poptart/.ssh/id_ed25519 type 3
debug1: identity file /home/poptart/.ssh/id_ed25519-cert type 7
[...]
debug1: Authenticating to hosakacorp.net:31337 as 'poptart'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519-cert-v01@openssh.com
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com SHA256:pdjY7SJknGheBqZCafzr+LuTf/sfLeXmqYCywJ2RwQU serial 2 ID "hosakacorp.net" CA ssh-ed25519 SHA256:a2qKtUqfuIskFdHSS+7RK8eX2p73F9kYp419+aQ2NBA valid forever
debug1: Host 'hosakacorp.net' is known and matches the ED25519-CERT host certificate.
debug1: Found CA key in /home/poptart/.ssh/known_hosts:20
[...]
debug1: Will attempt key: /home/poptart/.ssh/id_ed25519 ED25519 SHA256:Oz2vZgzxhyINBkG94gz5gk5aCIKiJaefP81NXwSTfI7 explicit agent
debug1: Will attempt key: /home/poptart/.ssh/id_ed25519 ED25519-CERT SHA256:Oz2vZgzxhyINBkG94gz5gk5aCIKiJaefP81NXwSTfI7 explicit agent
debug1: Will attempt key: poptart@altered.hosakacorp.net ED25519 SHA256:K1NKnlRN2lnQznQZ+tfu8kbkSIZudP9N7i4UEL0oK2U agent
[...]
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/poptart/.ssh/id_ed25519 ED25519 SHA256:Oz2vZgzxhyINBkG94gz5gk5aCIKiJaefP81NXwSTfI7 explicit agent
debug1: Authentications that can continue: publickey
debug1: Offering public key: /home/poptart/.ssh/id_ed25519 ED25519-CERT SHA256:Oz2vZgzxhyINBkG94gz5gk5aCIKiJaefP81NXwSTfI7 explicit agent
debug1: Server accepts key: /home/poptart/.ssh/id_ed25519 ED25519-CERT SHA256:Oz2vZgzxhyINBkG94gz5gk5aCIKiJaefP81NXwSTfI7 explicit agent
debug1: Authentication succeeded (publickey).
Authenticated to hosakacorp.net ([...]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
[...]
debug1: Remote: cert: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: principals: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Last login: Tue Feb 18 03:27:33 2020 from [...]
OpenBSD 6.6 (GENERIC.MP) #372: Sat Oct 12 10:56:27 MDT 2019
"The criminal is a creative artist; detectives are just critics."
perhonen$
One additional feature that is inherited by this system is that “approved” configuration options for the SSH keys can be embedded and signed directly into the certificates themselves. The exact options can be set with the -O options when signing the keys from the CA, but allow for third party extensions, client restrictions (such as removing the ability to do X forwarding or agent forwarding, or even explicitly allowing it in default deny environments), restricting source address, or forcing execution of specific commands.
In order to add restrictions here are some examples of options that can be set:
force-command="/var/www/deploy/do.sh" - Forces the signed keys to only be able to execute a single command. For all of my accounts that simply deploy or configure from CI this is what I prefer to do.no-port-forwarding - Disables port forwarding at the key level (reminder don’t rely on this for your only protection).source-address="10.0.1.0/24" - Deny keys not coming from the 10.0.1.0/24 subnet.In a large corporate deployment or one that is centrally managed there are quite a few special notes that need to be taken into account before adopting this process, or at least they should be considered. I wont go too much into detail about them, but they are important things that should be considered when using this functionality.
Probably the biggest elephant in the room for a system like this is the fact that instead of just adding a user to a LDAP configuration and publishing their SSH public keys for lookup there now needs to be some sort of process to send in keys, assign principals and identity, sign the keys, and give the keys back to the user. At the moment, this would most likely have to be an integrated onboarding process as well as a process to support when people inevitably lose or have keys stolen.
This also opens up the issue that the CA signing process will need some sort of automation as you don’t want to have users logging into the CA host directly if you can prevent it. This could be done by some sort of LDAP integrated automation or a separate process. I see this as a core adoption issue point and have been exploring potential options for creating CSR’s that are tied to identities in a more “enterprisy” fashion so stay tuned for my project on that.
Also worth noting is that keys expire. Unlike with a normal SSH configuration key validity is now something that needs to be monitored or understood at a process level, which also means key rotation and time synchronization.
Many systems have some sort of emergency “break glass” keys or credentials that are essentially admin everywhere, while this gives security purists a heart attack it does serve the purpose of never losing access during critical situations. Sometimes these can be dictated by standard (NIST SP 800-53) or by some other policy. If this becomes necessary, the authorized_keys file and AuthorizedKeysFile config can still be used alongside the PKI configuration. Just ensure that usage of that key is monitored and alerted heavily.
The CA keys should be treated as precious cargo and access to them in any sort of direct management scenario should be restricted, monitored, and periodically reviewed whenever used. That additionally means that a process should be created to ensure that they can be only recreated or accessed in a backup state by authorized parties. A mitigating feature that allows for more traditional protection of the CA keys and for additional backups would be to use a Hardware Security Module (HSM). The ssh-keygen(1) command has direct support for PKCS#11 that can be used to sign keys directly using the -D argument, but this is a bit of a tangent in it’s own and the only HSM I have direct access to does not support what I consider to be safe algorithms.
I don’t want to go into too much detail as it is out of scope for this document, but restricting the methods for authorization in a PKI environment and removing the user controlled key authorization is critical. So disabling PasswordAuthentication, GSSAPIAuthentication, and setting AuthorizedKeysFile to a admin controlled location for the above mentioned emergency situations (or have it in an admin controlled location and have that be empty when it isn’t needed). Reducing the amount of authentication methods in SSH and moving the authorization from the user control to the admin control reduces bad actors from dynamically adding keys for authorization without the admins approval.
From my experience in penetration testing I can’t stress how often I have gained some sort of remote write privilege on a web server or Jenkins runner and that keys are plainly available for me in a user home directory or some automatic SSH based deployment system attempts to log into a host I can simply log the incoming commands. By switching to this style of managed SSH configuration you protect yourself from various attacks and even better simplify some of the automation steps. Just ensure that your organization is fully aware and able to deal with the overhead of key management.
In other words, please stop using StrictHostKeyChecking=no.
Here are some scripts / templates that I use on for my daily KRL management:
#!/bin/sh -e
# krl is signed from CI using:
signify -Sz -s key-krl.sec -m /etc/ssh/ca/ssh_revoked_keys -x /srv/ssh_revoked_keys.gz && scp /srv/ssh_revoked_keys.gz ci:krl_inbound/ssh_revoked_keys.gz
signify -Sz -s key-ca.sec -m /etc/ssh/ca/ssh_ca_ed25519.pub -x /srv/ssh_ca_ed25519.pub.gz && scp /srv/ssh_ca_ed25519.pub.gz ci:ca_inbound/ssh_ca_ed25519.pub.gz
#!/bin/sh -e
# signify(1) key is distributed on all hosts on /etc/signify/krl-keytype.pub
ftp https://intra.hosakacorp.net/ssh_revoked_keys.gz | signify -Vz -t krl > /etc/ssh/ca/ssh_revoked_keys
ftp https://intra.hosakacorp.net/ssh | signify -Vz -t krl > /etc/ssh/ca/ssh_revoked_keys