Using Self-signed Certificates in Kubernetes

Use-Case

Sometimes you have an internal service or application that should be accessible only on your LAN.

Possible solutions:

• Access the service directly on its internal IP. However, this means exposing it on a non-standard port (since it is in Kubernetes, and ports 80 and 443 are occupied by the ingress controller). This makes for a messy URL for your users, and you don't get TLS (devices don't like certificates made out IP addresses instead of DNS names).
• Just expose it as a public service and restrict access it with an IP whitelist. But if you're running Kubernetes on bare metal there are potential issues with getting the correct external IP on incoming traffic. Also, I didn't manage to make a working setup where I could access the service over VPN - even with "send all traffic" enabled the IP detected by the ingress controller was the public IP of my device, not the VPN IP.
• Make a DNS entry pointing to the internal IP of your Kubernetes server. This means you can't use Let's Encrypt HTTP01 challenges, since Let's Encrypt can't reach the service from the outside. If your DNS provider doesn't have support for DNS01 integration (or you haven't / don't want to set that up), you can still get TLS with a self-signed certificate. This is what we'll do below.

Creating a Self-signed Certificate with a CA Certificate

What you strictly need to get TLS working for a service, is a server certificate made out to the DNS name where the service is accessible. However, getting a client device to trust a self-signed server certificate is kind of hard. A much better solution is to create a self-signed Certification Authority (CA) certificate, getting the client devices to trust that, and then using the CA certificate to sign a server certificate.

Create a CA Certificate

openssl genrsa -aes256 -out ca.key 4096

Make sure you store the generated key ca.key in a safe place, it is a private key and should not be exposed or shared with anyone.

Now, use the private key to create a CA certificate, valid for 10 years:

openssl req -new -key ca.key -x509 -out myca.crt -days 3650

Store the certificate myca.crt as well. It is not secret, but you need it available.

Create a Server Certificate

Now, let's use the CA certificate to create a server certificate.

We will assume the service you want to expose is on the url https://myservice.example.com.

First step is to create a Certificate Signing Request (CSR):

openssl req -new -nodes -newkey rsa:4096 -keyout myservice.key -out myservice.req -batch -subj "/C=SE/ST=MyRegion/L=MyCity/O=MyOrganization/OU=Internal/CN=myservice.example.com" -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]\nsubjectAltName=DNS:myservice.example.com"))

There is a bit to unpack here. In the subject you can put anything in the fields C (country code), ST (state/region), L (location/city), O (organization) and OU (organization unit/department).

CN must to be the DNS name of the service. It also must match the subjectAltName:DNS in the last part of the command.

Running this command will create two files, myservice.key and myservice.req.

Next step is to use the CSR (myservice.req) to create a service certificate, valid for one year:

openssl x509 -req -in myservice.req -CA myca.crt -CAkey ca.key -CAcreateserial -out myservice.crt -days 365 -sha256 -extfile <(printf "subjectAltName=DNS:myservice.example.com")

Again, it is important that subjectAltName=DNS matches the DNS name of the service. This creates a new file, called myservice.crt. You no longer need the req file.

Use the Server Certificate in Kubernetes

We will create an ingress, that uses the certificate we just created.

The ingress needs a secret, with the server certificate:

kubectl create secret -n mynamespace tls myservice-tls --cert myservice.crt --key myservice.key

With the secret in place, we can create the ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myservice-ingress
  namespace: myservice
  annotations:
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  ingressClassName: traefik
  tls:
    - hosts:
        - myservice.example.com
      secretName: myservice-tls
  rules:
    - host: myservice.example.com
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: myservice-service
                port:
                  number: 8080

The most important line here is secretName where we are referring to the secret with certificate we created in the previous step.

Making it work on devices

If you go to your service using a browser, you will get an error message saying the connection is insecure ("Untrusted certificate authority" or something along those lines). We need to make your client device trust your CA certificate. This has to be done on every user's device. Here are some examples on how to do it:

• MacOS (Chrome, Safari etc.): Open Keychain Access, go to System, then drag myca.crt into the list. Right click on the certificate in the list, expand the section Trust and change When using this certificate from System default to Always trust.
• Windows (Chrome, Safari, Edge etc.): Use this guide https://www.thewindowsclub.com/manage-trusted-root-certificates-windows
• Firefox (any desktop OS): Firefox has its own certificate store, so you add the CA cert to Firefox. Go to Settings, search for View certificates and on the tab Authorities press Import and import myca.crt
• iOS: Open myca.crt - either Airdrop it, send it as a message or download it from a url you've set up. You will be prompted you to install it in Settings. In Settings under General > VPN and devices your certificate will be shown under Retrieved profiles. Click it and install it. Finally, go back to General and then About and finally Settings for trusted certificates. Turn on Activate full trust for root certificate.

Troubleshooting

Everything involving certificates is extremely picky with everything being exactly correct. The most important parts are validity and names. Since you just created your certificates, it is unlikely your certificates aren't valid. If you are trying to use your certificate on iOS, it is important that the certificate isn't valid for too long - iOS will block certificates that are valid more than 390 days (that's why we chose 365 days above).

The CN of the server certificate's Subject field must match the DNS name of your service exactly (or match the pattern if it's a wildcard certificate). The Subject Alternative Name must also match the CN and the DNS name exactly. You can check those values of your server certificate using openssl:

openssl x509 -in myservice.crt -text -noout

Good luck!