Debugging Prometheus Scrape Issues

Prometheus is opaque and unhelpful for letting you know if it's even scraping your targets. This despite one of the touted features of Prometheus is that the scrape itself is a datapoint of whether a target is online. This post will walk you through a playbook for troubleshooting Prometheus scrape issues.

Steps

1. Verify the instance selects the scrape monitor

Prometheus does not have a strategy for scaling, and recommends using multiplicities of prometheus instances. This means that you have to actively manage which instance a scrape job (PodMonitor and ServiceMonitor) is assigned to.

Identify which Prometheus instance you want

Identify the relevant selectors, for example with:

kubectl get -A prometheuses -o json | jq '[ .items[] | {name: .metadata.name, namespace: .metadata.namespace, podNamespace: .spec.podMonitorNamespaceSelector, pod: .spec.podMonitorSelector, svcNamespace: .spec.serviceMonitorNamespaceSelector, svc: .spec.serviceMonitorSelector}]'

See if the selector is correct by running it manually. Here's an example for a matchlabel:
```
kubectl get ServiceMonitor -n prom --selector release==prom
```

2. Verify that Prometheus has picked up the monitor

Kubernetes is only eventually consistent, so maybe now isn't eventually. Manually check if Prometheus just needs more time to decide that it should check for service monitors. (There is no way to trigger this manually, so you'll just have to wait.)

port-forward to the Prometheus instance

kubectl port-forward -n prom pod/prometheus-prom-kube-prometheus-stack-prometheus-0 9090

go to the "Target Health" under "Status" tab (http://localhost:9090/targets) and check if the scrape job is there. If it's an empty pool, you might need to unhide it (http://localhost:9090/targets?pool=&health=unknown and click "Show empty pools")
If it isn't there, you can check the raw config (http://localhost:9090/config) and search through it. If it isn't in the config, but the selector from step 1 is correct, you'll have to wait (try killing pods lol).

3. Verify that the instance selector is working

Check that your monitor actually targets the objects you think it does. Maybe you have a typo in the selector.

Get the selectors

kubectl get ServiceMonitor -n prom prom-grafana -ojson | jq '{namespace: .spec.namespaceSelector, selector: .spec.selector}'

Manually simulate the selectors

kubectl get Service --namespace "prom" --selector app.kubernetes.io/instance==prom,app.kubernetes.io/name==grafana

4. Verify that the relabelling isn't discarding everything

Your target might be scraped, but the relabelling rules cause all of its metrics to be discarded. There's a handy simulator debugger for this integrated into Prometheus itself at the "Service discovery" under the "Status" tab (http://localhost:9090/service-discovery). Find your item and click "show relabeling"

I enjoy working with the Relabeler playground, which also helps you test rules until they work.