Advanced Nginx Ingress Configurations in Kubernetes

The ingress is a powerful tool in Kubernetes, since it dictates how resources are accessed from the outside. But it's also a tricky component to get right. Here are some things I've learned along the way.

Get the IP of connecting client

A problem I've struggled with is getting the IP of the connecting client into the ingress controller. Without it, things like IP whitelisting is impossible. For some reason, the default configuration is to show the IP of the connecting Kubernetes service.

How to fix

In the ingress-nginx-controller service, you need to make sure that externalTrafficPolicy: Local is set.

If you installed ingress-nginx with helm, you can add this to your values:‌‌

spec:
  values:
    controller:
      service:
        externalTrafficPolicy: Local

How to verify

Go to your ingress-nginx-controller deployment, and view the logs. The access log IPs should now be external IPs and not internal Kubernetes ones.

Separate configuration for subpath

What if you want to password protect or IP whitelist a path,  like /admin?

The biggest insight here is that you can have multiple ingresses for a service, as long as they don't have colliding prefix paths. So make one ingress for / that serves the public content, and a second ingress for /admin.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api
  namespace: production
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt"
spec:
  tls:
    - hosts:
        - api.example.com
      secretName: api-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api
  namespace: production
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt"
    nginx.ingress.kubernetes.io/whitelist-source-range: "192.168.0.0/16"
spec:
  tls:
    - hosts:
        - api.example.com
      secretName: api-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /admin
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 80

These two ingresses will coexist, and more specific paths like /admin will take precedence over more generic ones like /.

Mount a different service as a subpath

With multiple ingresses, you can even mount a different services as subpaths of the the same domain name. Adding to the example above, let's mount Kibana as api.example.com/logging. If the service is in the same namespace as the ingress, you can just put a different service name. But it the target service is in another namespace you need to add a new "alias service" in the namespace of the ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api
  namespace: production
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt"
    nginx.ingress.kubernetes.io/whitelist-source-range: "192.168.0.0/16"
spec:
  tls:
    - hosts:
        - api.example.com
      secretName: api-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /logging
            pathType: Prefix
            backend:
              service:
                name: elasticsearch-kibana
                port:
                  number: 5601
---
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-kibana
  namespace: production
spec:
  type: ExternalName
  externalName: elasticsearch-kibana.logging.svc.cluster.local

This adds a service called elasticsearch-kibana in the namespace production, that is just an alias a service also called elasticsearch-kibana in the namespace logging.

Kibana will now be accessible at the url https://api.example.com/logging, only accessible to clients in the IP range 192.168.*.*.

Note for this particular example: to make Kibana play nice in a subpath, you need to set the basePath variable in the Kibana configuration. If you are using bitnami's helm chart for ElasticSearch, add this to your values:

kibana:
  configuration:
    server:
      basePath: /logging
      rewriteBasePath: true

Wrap-up

Ingresses are your friends when configured right. They can often take care of tasks that could be potentially difficult to configure in underlying services. For instance, I needed to set up a container registry in a cluster and needed some kind of auth for it. Configuring auth in the registry container itself was a real hassle, but just slapping basic auth on it in the ingress solved all of my requirements. Same with Kibana.