Add Wildcard DNS Lookups in RKE2 Kubernetes with CoreDNS

Sometimes you need to add your own DNS entries to containers running in your Kubernetes cluster. One way of doing that is by adding hostAliases to your deployment:‌‌‌‌

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      hostAliases:
        - ip: "192.168.1.100"
          hostnames:
            - "foo.local"
            - "bar.local"

But what if you want to do this for all containers in your whole cluster? One use case for this is when your nodes cannot reach themselves on their external IPs (because of network/firewall configurations outside of the cluster). When this happens, for instance the Let's Encrypt issuer will fail since it always does an internal dry run on the hostname.

CoreDNS to the rescue! I've previously described how to add individual DNS entries to CoreDNS when you are running K3S. If you are using Rancher's other Kubernetes distribution RKE - the big brother of K3S - you need to add entries in a slightly different way. But the upside is you can use it to not only add individual entries, but also wildcard mappings like *.company.com.

You make changes to CoreDNS in RKE2 by adding a HelmChartConfig, that will change the already existing CoreDNS Helm release that RKE2 installs by default.

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-coredns
  namespace: kube-system
spec:
  valuesContent: |-
    servers:
    - zones:
      - zone: .
      port: 53
      plugins:
      - name: template
        parameters: "IN A dev.company.com"
        configBlock: |
          match ^([a-z0-9-]+)\.dev\.company\.com\.$
          answer "{{ .Name }} 60 IN A 192.168.1.123"
          fallthrough
      - name: errors
      - name: health
        configBlock: |-
          lameduck 5s
      - name: ready
      - name: hosts
        configBlock: |-
          192.168.1.123 test.company.com
          fallthrough
      - name: kubernetes
        parameters: cluster.local in-addr.arpa ip6.arpa
        configBlock: |-
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
          ttl 30
      - name: prometheus
        parameters: 0.0.0.0:9153
      - name: forward
        parameters: . /etc/resolv.conf
      - name: cache
        parameters: 30
      - name: loop
      - name: reload
      - name: loadbalance

What this file will do when applied, is change the Helm release rke2-coredns to include one CoreDNS template that will map *.dev.company.com to 192.168.1.123, and a specific host entry mapping test.company.com to the same IP. After this is applied, any DNS lookups to these names will resolve to the IP specified, in any container inside the cluster.

The way to resolve the specific problem with Acme challenge preflights is to map any hostnames you want to use with Let's Encrypt to an internal IP of any of your nodes that have an ingress controller running. It's not a particularly pretty solution, but it's better than not getting any certificates issued.