Skip to main content

Prometheus Query

Check instance and job up time​

up
--output--
up{instance="mail-server", job="01-server-cadvisor"} 0
up{instance="db-server", job="02-server-node-exporter"} 1
up{instance="app-server", job="03-server-cadvisor"} 1
up{instance="ob-server", job="04-prometheus"}
up{instance="https://dev.api.xxx.com", job="httpd"} 1
up{instance="https://dev.assets.xxxx.com", job="httpd"} 1
up{instance="172.26.14.199:9090", job="blackbox-prometheus"}
  • 0 = down / 1 = up
  • Example fillter down "up == 0"

Specific instance check up/down​

  up{instance="ob-server", job="ob-prometheus"}

List all instances & jobs​

node_disk_info # List all instances and jobs
--output--

node_disk_info{device="nvme0n1", instance="01-server", job="01-server-node-exporter", major="259", minor="0"} 1
node_disk_info{device="nvme0n1", instance="02-server", job="02-server-node-exporter", major="259", minor="0"} 1
node_disk_info{device="nvme0n1", instance="03-server", job="03-server-node-exporter", major="259", minor="0"} 1
node_disk_info{device="nvme0n1", instance="04-server", job="04-server-node-exporter", major="259", minor="0"} 1
node_disk_info{device="nvme0n1", instance="05-service-server", job="05-service-node-exporter", major="259", minor="0"} 1
node_disk_info{device="nvme0n1", instance="06-server", job="06-server-node-exporter", major="259", minor="0"} 1
node_disk_info{device="nvme0n1", instance="07-server", job="07-server-node-exporter", major="259", minor="0"} 1
node_disk_info{device="nvme0n1", instance="08-server", job="08-server-node-exporter", major="259", minor="0"} 1
node_disk_info{device="nvme0n1", instance="09-server", job="09-service-node-exporter", major="259", minor="0"}

Selector and matchers​

prometheus_http_requests_total #
---output---
prometheus_http_requests_total{code="200", handler="/favicon.ico", instance="ob-server", job="ob-prometheus"} 0
prometheus_http_requests_total{code="200", handler="/federate", instance="ob-server", job="ob-prometheus"} 0
prometheus_http_requests_total{code="200", handler="/flags", instance="ob-server", job="ob-prometheus"} 0
prometheus_http_requests_total{code="200", handler="/graph", instance="ob-server", job="ob-prometheus"} 0
prometheus_http_requests_total{code="200", handler="/manifest.json", instance="ob-server", job="ob-prometheus"} 0

## Need query matcher fiilltering /graph only
prometheus_http_requests_total{handler="/graph"}
---output---
prometheus_http_requests_total{code="200", handler="/graph", instance="ob-server", job="ob-prometheus"} 0

## wildcard matcher {handler=~"/wildcard or regex"}
prometheus_http_requests_total{handler=~"/api/v1/.*"} # .* wildcard matcher for regex
--output--
prometheus_http_requests_total{code="200", handler="/api/v1/*path", instance="ob-server", job="ob-prometheus"} 0
prometheus_http_requests_total{code="200", handler="/api/v1/admin/tsdb/clean_tombstones", instance="ob-server", job="ob-prometheus"} 0
prometheus_http_requests_total{code="200", handler="/api/v1/admin/tsdb/delete_series", instance="ob-server", job="ob-prometheus"} 0
prometheus_http_requests_total{code="200", handler="/api/v1/admin/tsdb/snapshot", instance="ob-server", job="ob-prometheus"} 0
  • "." in regex means any character or az, AZ, 0-9
  • "*" in regex means any number of characters or 0 to n

not equal matcher​

prometheus_http_requests_total{handler!~"/api/v1/.*"}
---output---
prometheus_http_requests_total{code="200", handler="/", instance="ob-server", job="ob-prometheus"} 0
prometheus_http_requests_total{code="200", handler="/-/healthy", instance="ob-server", job="ob-prometheus"} 0
prometheus_http_requests_total{code="200", handler="/-/quit", instance="ob-server", job="ob-prometheus"}0
prometheus_http_requests_total{code="200", handler="/-/ready", instance="ob-server", job="ob-prometheus"} 0

Need 2 matchers​

prometheus_http_requests_total{handler=~"/graph|/metrics"}
---output---
prometheus_http_requests_total{code="200", handler="/graph", instance="ob-server", job="ob-prometheus"} 0
prometheus_http_requests_total{code="200", handler="/metrics", instance="ob-server", job="ob-prometheus"} 101

Need http requests with status code 200 - 300​

prometheus_http_requests_total{code=~"2.*|3.*"}
---output---
prometheus_http_requests_total{code="200", handler="/", instance="ob-server", job="ob-prometheus"}
0
prometheus_http_requests_total{code="200", handler="/-/healthy", instance="ob-server", job="ob-prometheus"}
0
``
## except 4xx
```bash
prometheus_http_requests_total{code!~"4.*"}

---output---

More queries​

scrape_duration_seconds # Time taken to scrape the target