thoughts/data/ssh-ca-proxyjump.md

Key Takeaways

• SSH has a key-signing concept that in combination with a smartcard provides a lean, off-disk process
• A SSH-CA provides the possibility of managing access without a central point of failure
• The use of SSH Jumphost is an easier way to tunnel sessions end-to-end encrypted, while still maintaining visibility and control through a central point

Introduction

This post is an all-in-one capture of my recent discoveries with SSH. It is an introduction for a technical audience.

It turns out that SSH is ready for a zero trust and microsegmentation approach, which is important for management of servers. Everything described in this post is available as open source software, but some parts require a smartcard or two, such as a Yubikey (or a Nitrokey if you prefer open source. I describe both).

I also go into detail on how to configure the CA key without letting the key touch the computer, which is an important principle.

The end-result should be a more an architecture providing a better overview of the infrastructure and a second logon-factor independent of phones and OATH.

SSH-CA

My exploration started when I read a 2016-article by Facebook engineering [1]. Surprised, but concerned with the configuration overhead and reliability I set out to test the SSH-CA concept. Two days later all my servers were on a new architecture.

SSH-CA works predictably like follows:

                                       [ User generates key on Yubikey ]
                                                        |
                                                        |
                                                        v
[ ssh-keygen generates CA key ] --------> [ signs pubkey of Yubikey ]
                |                           - for a set of security zones
                |                           - for users
                |                                       |
                |                                       |
                |                                       v
                v                         pubkey cert is distributed to user
[ CA cert and zones pushed to servers ]     - id_rsa-cert.pub
  - auth_principals/root (root-everywhere)
  - auth_principals/web (zone-web)

The commands required in a nutshell:

# on client
$ ssh-keygen -t rsa

# on server
$ ssh-keygen -C CA -f ca
$ ssh-keygen -s ca -I <id-for-logs> -n zone-web -V +1w -z 1 id_ecdsa.pub

# on client
cp id_ecdsa-cert.pub ~/.ssh/

Please refer to the next section for a best practice storage of your private key.

On the SSH server, add the following to the SSHD config:

TrustedUserCAKeys /etc/ssh/ca.pub
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u

What was conceptually new for me was principals and authorization files per server. This is how it works:

1. Add a security zone, like zone-web, during certificate signing - "ssh-keygen * -n zone-web *". Local username does not matter
2. Add a file per user on the SSH server, where zone-web is added where applicable - e.g. "/etc/ssh/auth_principals/some-user" contains "zone-web"
3. Login with the same user as given in the zone file - "ssh some-user@server"

This is the same as applying a role instead of a name to the authorization system, while something that IDs the user is added to certificate and logged when used.

This leaves us with a way better authorization and authentication scheme than authorized_keys that everyone uses. Read on to get the details for generating the CA key securely.

Keeping Private Keys Off-disk

An important principle I have about private keys is to rather cross-sign and encrypt two keys than to store one on disk. This was challenged for the SSH-CA design. Luckily I found an article describing the details of PKCS11 with ssh-keygen [2]:

If you're using pkcs11 tokens to hold your ssh key, you may need to run ssh-keygen -D $PKCS11_MODULE_PATH ~/.ssh/id_rsa.pub so that you have a public key to sign. If your CA private key is being held in a pkcs11 token, you can use the -D parameter, in this case the -s parameter has to point to the public key of the CA.

Yubikeys on macOS 11 (Big Sur) requires the yubico-piv-tool to provide PKCS#11 drivers. It can be installed using Homebrew:

$ brew install yubico-piv-tool
$ PKCS11_MODULE_PATH=/usr/local/lib/libykcs11.dylib

Similarly the procedure for Nitrokey are:

$ brew cask install opensc
$ PKCS11_MODULE_PATH=/usr/local/lib/opensc-pkcs11.so

Generating a key on-card for Yubikey:

$ yubico-piv-tool -s 9a -a generate -o public.pem

For the Nitrokey:

$ pkcs11-tool -l --login-type so --keypairgen --key-type RSA:2048

Using the exported CA pubkey and the private key on-card a certificate may now be signed and distributed to the user.

$ ssh-keygen -D $PKCS11_MODULE_PATH -e  > ca.pub

$ ssh-keygen -D $PKCS11_MODULE_PATH -s ca.pub -I example -n zone-web \
             -V +1w -z 1 id_rsa.pub
Enter PIN for 'OpenPGP card (User PIN)': 
Signed user key .ssh/id_rsa-cert.pub: id "example" serial 1 for zone-web 
valid from 2020-10-13T15:09:00 to 2020-10-20T15:10:40

The same concept goes for a user smart-card, except that is a plug and play as long as you have the gpg-agent running. When the id_rsa-cert.pub (the signed certificate of e.g. a Yubikey) is located in ~/.ssh, SSH will find the corresponding private key automatically. The workflow will be something along these lines:

[ User smartcard ] -----------> [ CA smartcard ]
         ^          id_rsa.pub          |
         |                              | signs
         |------------------------------|
           sends back id_rsa-cert.pub

A Simple Bastion Host Setup

The other thing I wanted to mention was the -J option of ssh, ProxyJump.

ProxyJump allows a user to confidentially, without risk of a man-in-the-middle (MitM), to tunnel the session through a central bastion host end-to-end encrypted.

Having end-to-end encryption for an SSH proxy may seem counter-intuitive since it cannot inspect the content. I believe it is the better option due to:

• It is a usability compromise, but also a security compromise in case the bastion host is compromised.
• Network access and application authentication (and even authorization) goes through a hardened point.
• In addition the end-point should also log what happens on the server to a central syslog server.
• A bastion host should always be positioned in front of the server segments, not on the infrastructure perimeter.

A simple setup looks like the following:

[ client ]  ---> [ bastion host ] ---> [ server ]

Practically speaking a standalone command will look like follows:

ssh -J jump.example.com dest.example.com

An equivalent .ssh/config will look like:

 Host j.example.com
 HostName j.example.com
 User sshjump
 Port 22

 Host dest.example.com
 HostName dest.example.com
 ProxyJump j.example.com
 User some-user
 Port 22

With the above configuration the user can compress the ProxyJump SSH-command to "ssh dest.example.com".

Further Work

The basic design shown above requires one factor which is probably not acceptable in larger companies: someone needs to manually sign and rotate certificates. There are some options mentioned in open sources, where it is normally to avoid having certificates on clients and having an authorization gateway with SSO. This does however introduce a weakness in the chain.

I am also interested in using SSH certificates on iOS, but that has turned out to be unsupported in all apps I have tested so far. It is however on the roadmap of Termius, hopefully in the near-future. Follow updates on this subject on my Honk thread about it [4].

For a smaller infrastructure like mine, I have found the manual approach to be sufficient so far.

[1] Scalable and secure access with SSH: https://engineering.fb.com/security/scalable-and-secure-access-with-ssh/
[2] Using a CA with SSH: https://www.lorier.net/docs/ssh-ca.html
[3] Using PIV for SSH through PKCS #11: https://developers.yubico.com/PIV/Guides/SSH_with_PIV_and_PKCS11.html
[4] https://cybsec.network/u/tommy/h/q1g4YC31q45CT4SPK4