Monitoring Kubernetes Ingress with BlackboxExporter
BlackboxExporter is a standard tool used to monitor different endpoints with Prometheus. Many examples that you find on the internet will show you how to monitor a static list of endpoints. While a static list is easy to use in small deployments, it is very easy to forget about updating that list once you have multiple clusters or adding new Ingress resources to your clusters often. Maintenance of such monitoring solution becomes cumbersome and painful quite fast.
Prometheus provides a discovery mechanism for Kubernetes resources, including Ingress. Let’s see how the job would look:
This job covers a simple use case where your Ingress resources are separate domains with a root path for scraping like https://test.example.com/. If you delete the first rule, then tests would be done with an HTTP endpoint, however, this does not always work well. Without HTTPS request, you will not know if your domain certificate has expired, for instance. Blackbox exporter provides a simple UI where you can see the results of each test, and concrete steps during the test such as DNS resolution and redirection of your requests. You can check it out via port-forwarding:
kubectl port-forward svc/blackbox-exporter 9115
And then you can drill down to the debug logs:
So, with a single query, you know at least the following things:
- DNS is working
- Domain certificate hasn’t expired
- The load balancer is working (or NodePort is open in your firewall)
- Network policies are not blocking ingress traffic (assuming you are using them)
- Kubernetes service is properly configured
- Kubernetes pod is up and healthy (assuming your liveness and readiness probes do a proper job on validating your application health)
If you have a more sophisticated setup with Ingress, you may have to tune the job a bit, with regards to relabel_configs. How do you figure out which data is accessible to Prometheus? You can check the discovery tab of Prometheus:
Then click on the Blackbox job to see all discovered Ingress resources and available labels:
Based on available labels, your Prometheus job could look like this:
The only problem is that __meta_kubernetes_ingress_scheme always returns HTTP on our clusters. This may be related to the fact that we are managing certificates outside of Kubernetes. So, I would still either hardcode the protocol to HTTPS or add an extra label to the Ingress resource with the protocol specification (if certificates are managed separately).
Finally, you may wonder what to do if you are using some other Ingress controllers, like Istio that have their own API spec. You have a few options:
- Make a merge request to the Prometheus repository to support discovery of that particular API.
- Go back to a static list of servers, as shown in most examples.
- Use file discovery.
The last method could work with an extra app and a shared volume between Prometheus and the extra app responsible for ingress discovery. Your app can list your Istio gateways and write them to a file in a proper format. Prometheus will pick up changes to the static list of endpoints from that file and probe your endpoints via BlackboxExporter.