A pretty common setup is when you want an internal web site or web application served on 192.168.x.x. or similar LAN IP, but with a regular Let's Encrypt certificate. If you have a DNS provider with existing kubernetes support (i.e. a cert-manager hook) for DNS01 (like Amazon Route 53) you can stop reading - you're good to go. But what if you don't?
If your DNS provider doesn't have kubernetes support for DNS01 you do have a problem. You are likely using HTTP01 for your external sites, which is fine. But you can't do that with internal sites, because they are not reachable from the outside.
Solution: Run your own DNS server.
This is a scary concept for most. You usually don't want to assume the responsibility for DNS lookups into your kubernetes setup, just like most people don't want to run their own mail server. But here's the trick - only do it for a subdomain where you host your internal sites. So let contoso.com be handled by an external provider, but delegate internal.contoso.com to your self-hosted DNS. This is a much more stable and automatic solution than using self-signed certificates.
How to
This is going to be quite a long one, but a lot of it is kubernetes manifests. It's easier than it looks and it's going to be worth it.
The setup consists of four steps:
- A BIND9 DNS server running in Kubernetes, serving
internal.contoso.comonly. BIND9 is made by the Internet Systems Consortium themselves, you can't go more standard than that - Making that DNS server reachable from the internet on port 53
- Delegate
internal.contoso.comto your DNS server at your external DNS provider - 4. A cert-manager ClusterIssuer that uses the built-in RFC2136 solver to create the DNS01 challenge records
RFC2136 is the standard for dynamic DNS updates, and cert-manager supports it out of the box. You don't need a third-party webhook.
We will assume your ingress controller (Traefik in my case) has the LAN IP 192.168.1.10, and that your public (static) IP is 203.0.113.10.
Create a TSIG Key
cert-manager signs its updates to BIND with a shared secret called a TSIG key. Generate one with tsig-keygen, which comes with BIND (brew install bind on MacOS):
tsig-keygen -a hmac-sha512 cert-managerThe output looks like this:
key "cert-manager" {
algorithm hmac-sha512;
secret "<long base64 string>";
};Keep it somewhere safe. Anyone who has it can change records in your zone.
The DNS Server
First, prerequisites - namespace, TSIG key and BIND config:
apiVersion: v1
kind: Namespace
metadata:
name: dns
---
apiVersion: v1
kind: Secret
metadata:
name: dns-tsig
namespace: dns
type: Opaque
stringData:
secret: |
key "cert-manager" {
algorithm hmac-sha512;
secret "<long base64 string>";
};
---
apiVersion: v1
kind: ConfigMap
metadata:
name: dns-server-config
namespace: dns
data:
named.conf: |
options {
directory "/var/lib/bind";
recursion no;
allow-recursion { none; };
allow-query { any; };
allow-transfer { none; };
listen-on { any; };
listen-on-v6 { any; };
rate-limit {
responses-per-second 50;
window 5;
};
dnssec-validation no;
version none;
};
include "/etc/bind/tsig.key";
zone "internal.contoso.com" {
type primary;
file "/var/lib/bind/internal.contoso.com.zone";
allow-update { key "cert-manager"; };
};
logging {
channel default_log {
stderr;
severity info;
print-time yes;
};
category default { default_log; };
};
internal.contoso.com.zone.seed: |
$TTL 300
@ IN SOA ns1.internal.contoso.com. hostmaster.internal.contoso.com. (
2026101600 ; serial
3600 ; refresh
900 ; retry
604800 ; expire
300 ) ; negative-cache TTL
IN NS ns1.internal.contoso.com.
ns1 IN A 203.0.113.10
myservice IN A 192.168.1.10A few things worth pointing out:
recursion nois super important! This makes the DNS server only answers questions aboutinternal.contoso.comand nothing else. An open recursive resolver on the internet gets abused for DDoS amplification within days.allow-updateonly allows changes signed with your TSIG key.- The rate limit is generous on purpose. During issuance, the Let's Encrypt validators and cert-manager's own checks send bursts of queries. I had Google's 8.8.8.8 refuse the entire zone for a while once, after a lot of certificate churn with a stricter limit.
- The zone file contains your internal records, like
myservice. Yes, this means anyone can look upmyservice.internal.contoso.comand see that it points to 192.168.1.10. That's the whole point (otherwise Let's Encrypt won't work), and a private IP is useless from the outside anyway.
Now for the deployment. BIND writes dynamic updates to its zone file, so the zone has to live on a persistent volume. The ConfigMap only holds a seed. An init container copies it to the volume on first boot only, so the records and serial numbers from later updates survive restarts.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dns-server-data
namespace: dns
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dns-server
namespace: dns
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: dns-server
template:
metadata:
labels:
app: dns-server
spec:
initContainers:
- name: seed-zone
image: busybox:1.36
command:
- sh
- -c
- |
dest="/var/lib/bind/internal.contoso.com.zone"
[ -f "$dest" ] || cp /etc/bind/seed/internal.contoso.com.zone.seed "$dest"
volumeMounts:
- name: data
mountPath: /var/lib/bind
- name: config
mountPath: /etc/bind/seed
containers:
- name: bind9
image: internetsystemsconsortium/bind9:9.20
command: ["named"]
args: ["-c", "/etc/bind/named.conf", "-g", "-f", "-p", "5353"]
ports:
- name: dns-tcp
containerPort: 5353
protocol: TCP
- name: dns-udp
containerPort: 5353
protocol: UDP
volumeMounts:
- name: config
mountPath: /etc/bind/named.conf
subPath: named.conf
- name: tsig
mountPath: /etc/bind/tsig.key
subPath: tsig.key
readOnly: true
- name: data
mountPath: /var/lib/bind
volumes:
- name: config
configMap:
name: dns-server-config
- name: tsig
secret:
secretName: dns-tsig
items:
- key: secret
path: tsig.key
defaultMode: 0400
- name: data
persistentVolumeClaim:
claimName: dns-server-data
---
apiVersion: v1
kind: Service
metadata:
name: dns-server
namespace: dns
spec:
selector:
app: dns-server
ports:
- name: dns-tcp
port: 5353
protocol: TCP
- name: dns-udp
port: 5353
protocol: UDPBIND listens on 5353 instead of 53, so the container itself doesn't need privileges to bind a low port. The ingress controller handles port 53 instead.
Note that the zone seed is only used on first boot. If you add static records later, either use nsupdate with the TSIG key or delete the zone file on the volume and restart the pod.
Exposing Port 53
I already have Traefik on 192.168.1.10 handling ports 80 and 443, so I let it handle DNS as well instead of setting up another LoadBalancer. Add two entrypoints to your Traefik Helm values:
ports:
dnstcp:
port: 53
protocol: TCP
exposedPort: 53
expose:
default: true
dnsudp:
port: 53
protocol: UDP
exposedPort: 53
expose:
default: trueAnd route them to the DNS server:
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
name: dns-tcp
namespace: dns
spec:
entryPoints:
- dnstcp
routes:
- match: HostSNI(`*`)
services:
- name: dns-server
port: 5353
---
apiVersion: traefik.io/v1alpha1
kind: IngressRouteUDP
metadata:
name: dns-udp
namespace: dns
spec:
entryPoints:
- dnsudp
routes:
- services:
- name: dns-server
port: 5353Finally, set up port forwarding for TCP and UDP port 53 on your router to 192.168.1.10.
Delegating the Subdomain
Log in to your DNS provider and add two records to contoso.com:
internal NS ns1.internal.contoso.com.
ns1.internal A 203.0.113.10The NS record tells the world that internal.contoso.com is handled by ns1.internal.contoso.com. The A record (a "glue" record) tells the world where that server actually is. Without it, nobody could find your name server, since its own name is inside the zone it serves.
Some providers don't let you create NS records in their web interface. Mine (Loopia) doesn't, but it was possible through their API. Check your provider's API documentation if you run into this.
Verify the delegation from outside your network, for example from your phone's hotspot:
dig myservice.internal.contoso.com @8.8.8.8If you get 192.168.1.10 back, everything works so far.
The ClusterIssuer
cert-manager reads the TSIG secret from its own namespace, so it needs a copy there. Note that cert-manager only wants the base64 string, not the whole key { ... } block we gave BIND:
kubectl -n cert-manager create secret generic dns-tsig --from-literal=secret=<long base64 string>Now create the issuer. Start with staging so you don't hit Let's Encrypt's rate limits while you get things working:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-internal-staging
spec:
acme:
email: you@contoso.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-internal-staging-key
solvers:
- dns01:
rfc2136:
nameserver: 192.168.1.10:53
tsigKeyName: cert-manager
tsigAlgorithm: HMACSHA512
tsigSecretSecretRef:
name: dns-tsig
key: secret
selector:
dnsZones:
- "internal.contoso.com"cert-manager creates a TXT record _acme-challenge.myservice.internal.contoso.com on your BIND server, Let's Encrypt looks it up via the public delegation, and a few minutes later you have a real certificate for a site that's only reachable on your LAN. No self-signed CA and no installing certificates on every device.
Troubleshooting
If the certificate is stuck, start with kubectl describe challenge -A, which usually tells you what is wrong. The most common problems:
- TSIG errors (`BADKEY`, `BADSIG`) in the BIND logs. The key name, algorithm or secret don't match between BIND and the ClusterIssuer. Check that the cert-manager secret contains only the base64 string.
- cert-manager's propagation check times out. Before telling Let's Encrypt to validate, cert-manager checks that the TXT record is visible on the authoritative name server. Inside the cluster this means connecting to your public IP, which won't work if your router doesn't support hairpin NAT. You can fix this by forwarding
internal.contoso.comto 192.168.1.10 in CoreDNS, the same way I described in [Add Local DNS Entries to K3S](https://metalcoder.dev/add-custom-dns-entries-to-k3s/). As a bonus, your pods can resolve your internal names too. - It works from inside your network but not from outside. Check the port forwarding, and remember UDP. Test with
dig @203.0.113.10 internal.contoso.com SOAfrom outside your network.
OK, that was a lot of work admittedly. Wouldn't it be easier to just use Route53? Well of course, but where's the fun in that? And we're trying to leave the tech bro oligarchy, remember?
Good luck!