PromQL

PromQL is Prometheus’ query language: pick a time series by name and labels, transform it with functions, and aggregate across the whole fleet. Here’s every query you’ll actually write.

Quick reference

The ten queries you’ll reach for every day. Each links to the full section below.

• rate(metric[5m]) — Per-second rate of a counter, reset-aware. Rates →
• irate(metric[5m]) — Per-second rate from the last two samples — for alerting. Rates →
• increase(metric[1h]) — Total counter increase over the window. Rates →
• sum by (label) (metric) — Aggregate, keeping the listed labels only. Aggregation →
• histogram_quantile(0.99, rate(bucket[5m])) — Percentile latency from histogram buckets. Functions →
• topk(5, metric) — Largest 5 series by value; bottomk for smallest. Aggregation →
• metric{job="api", status=~"5.."} — Label selectors & matchers. Selectors →
• sort(metric) / sort_desc(metric) — Order series by value. Functions →
• rate(metric[5m]) / rate(metric[5m] offset 1h) — Compare now vs an hour ago. Functions →
• up == 0 — Targets that are down. Operators →

Metric model

Every metric is a name plus a set of labels. A time series is that combination sampled over time.

Metric name (http_requests_total) → Labels ({job=“api”, status=“200”}) → Time series (name + label set) → Sample (value @ timestamp) → Vector (instant vs range)

1. Name

All series with that metric name.

http_requests_total

2. Filter

Narrow to a label set.

http_requests_total{job="api"}

3. Window

Turn it into a range vector.

http_requests_total[5m]

4. Rate

Compute per-second change.

rate(http_requests_total[5m])

A sample

The atomic unit: a float64 value plus a millisecond timestamp.

http_requests_total{job="api"} 10421 1700000000000

Instant vector

One sample per series at a single point in time — what most expressions return.

http_requests_total
{job="api"}

Range vector

Many samples per series across a window — what rate() and friends consume.

http_requests_total[5m]
node_cpu_seconds_total[1h]

KEY: Labels define identity. Every unique label set — http_requests_total{job="api", status="200"} — is its own time series. Adding a label multiplies the number of series, so keep cardinality low.

Selectors & matchers

Select a subset of series by matching on label values. The metric name is itself a label.

Metric name is a selector

The name is just a label. http_requests_total is shorthand for {__name__="http_requests_total"}.

http_requests_total
{__name__="http_requests_total"}

Regex is fully anchored

=~ matches the whole value, so job=~"api" matches only api — not apiserver. Add .* to widen.

up{job=~"api.*"}
up{env!~"dev|staging"}

up{job=~"api.*"}
up{job=~".*prod"}

up{env!~"dev|staging|test"}

{__name__=~"http_.*_total"}

up{job!=""}

KEY: Commas AND, braces OR. Multiple matchers inside one {…} must all hold; separate selectors separated by commas are independent series.

Rates & counters

Counters only ever go up (until a reset), so query their rate of change — never their raw value.

1. rate()

Per-second average over the window. Extrapolates the edges and handles counter resets. Best for dashboards.

rate(http_requests_total[5m])

2. irate()

Per-second rate from the last two samples only. Reacts faster but is noisier. Best for alerting.

irate(http_requests_total[5m])

3. increase()

Total increase over the window, reset-aware. Use when you want a count, not a rate.

increase(http_requests_total[1h])

4. delta()

Difference over the window for gauges. Does not handle counter resets.

delta(node_memory_available_bytes[1h])

rate(http_requests_total[5m])   # dashboards
irate(http_requests_total[5m])  # alerts

KEY: Rate first, then sum. Apply rate() to each counter series before aggregating: sum(rate(http_requests_total[5m])). Summing raw counters first hides resets and double-counts when series churn.

Aggregation

Aggregation operators collapse many series into fewer by grouping on a chosen label set.

by vs without

by (…) keeps only the listed labels; without (…) drops the listed labels and keeps everything else.

sum by (job, status) (rate(x[5m]))
sum without (instance) (rate(x[5m]))

topk & bottomk

topk(k, v) returns the k series with the largest values in v; bottomk the smallest.

topk(3, rate(x[5m]))
bottomk(3, node_load1)

⚠: Aggregation quantile is across series, not a histogram percentile. For latency percentiles use histogram_quantile() with bucket series.

count_values("version", build_info)
count by (app) (up)     # instances per app
group by (job) (up)     # one series per job

Functions

The workhorses beyond rate and aggregation — histograms, sorting, prediction, and label surgery.

• histogram_quantile(0.99, rate(bucket[5m])) — 99th-percentile latency from histogram buckets.
• sort(v) / sort_desc(v) — Sort series by value ascending / descending.
• predict_linear(v[1h], 4*3600) — Linear forecast 4h ahead (disk-full alerts).
• absent(v) — 1 if v has no series, empty otherwise.
• absent_over_time(v[5m]) — 1 if v had no samples in the window.
• time() — Current Unix timestamp in seconds.
• v offset 1h — Shift the query one hour into the past.
• label_replace(v, "dst", "$1", "src", "(.*)") — Rewrite a label using a regex capture.
• clamp(v, 0, 100) — Clamp every value into [0, 100].
• sort_by_label(v, "label") — Sort series by a label’s value.
• last_over_time(v[5m]) — Last sample in the window — the right way to read a gauge’s current value.
• info(v, {__name__="build_info"}) — Experimental (3.x): enrich series with labels from target_info/build_info — needs --enable-feature=promql-experimental-functions.

Histogram quantiles

Always pass the bucket series through rate() first and keep the le label intact.

histogram_quantile(0.99,
  rate(http_request_duration_seconds_bucket[5m]))

offset for comparisons

offset shifts a query back in time so you can compare now against then.

(rate(x[5m]) - rate(x[5m] offset 1h))
  / rate(x[5m] offset 1h)

histogram sorting prediction absence labels time metadata experimental

label_replace(v, dst, replacement, src, regex)

label_replace(up, "instance", "$1",
  "instance", "(.*):9100")

absent(nonexistent_metric)

absent_over_time(up[5m])

Operators & vector matching

Arithmetic and comparison work element-wise across series; set operators combine whole vectors.

on / ignoring

Control which labels must match when joining two vectors. on lists labels to join on; ignoring lists labels to ignore.

a / on (instance) b
a / ignoring (job) b

group_left / group_right

Many-to-one joins. group_left keeps the extra labels on the left-hand side.

rate(http_errors_total[5m])
  / ignoring (code) group_left
    rate(http_requests_total[5m])

KEY: No matching labels → empty result. Two vectors only pair up on series with identical label sets (modulo on/ignoring). If nothing matches, you get no output — silently, not an error.

up == 1 and rate(http_requests_total[5m]) > 0  # up AND receiving traffic
rate(node_errors_total[5m]) or vector(0)          # never empty
up unless up{job="ignored"}                       # drop one job

up == 1            # only the up series
up == bool 1       # every series as 0/1
rate(x[5m]) > bool 0.5

Recording rules & alerts

Recording rules precompute expressions into new metrics; alerting rules evaluate a condition and fire.

Recording rule

Store an expensive expression under a new metric name, updated every evaluation interval.

- record: job:http_requests:rate5m
  expr: sum by (job) (rate(http_requests_total[5m]))

Alerting rule

Fires after expr holds true continuously for the for duration.

- alert: HighErrorRate
  expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
  for: 5m
  labels:
    severity: page
  annotations:
    summary: "errors on {{ $labels.job }}"

1. Evaluate expr — The server evaluates each rule’s expr on the rule group’s interval (default evaluation_interval).
2. Hold for — An alerting rule must stay true for the whole for duration before it transitions to pending.
3. Fire — The alert enters firing state and is sent to Alertmanager.
4. Route & notify — Alertmanager deduplicates, groups, routes, and sends notifications.

KEY: Name recording rules level:metric:operation — e.g. job:http_requests:rate5m. Rules are evaluated by the server (default evaluation_interval = 1m), so don’t record a raw metric you can already query directly.

groups:
  - name: http.rules
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - alert: HighErrorRate
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (job) (rate(http_requests_total[5m])) > 0.1
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} error rate above 10%"

Pitfalls

Small behaviors that silently change what a query returns.

Counter resets

A counter resets to 0 on restart. rate(), irate(), and increase() detect and handle resets; delta() does not.

rate(restarts_total[5m])  # safe
delta(restarts_total[5m]) # wrong

Window ≥ 2× scrape

Use a range at least twice the scrape interval so rate() has ≥2 samples. Too short a window extrapolates wildly.

# scrape_interval: 15s → window ≥ 30s
rate(http_requests_total[1m])

Label cardinality

Every unique label set is a separate time series. Never put user IDs, request paths, or other unbounded values in labels — use a histogram or log instead.

http_requests_total{path="/users/123"}  # bad
http_requests_total{route="/users/:id"}  # good

Staleness

A series that stops scraping goes stale after ~5m and its last sample is ignored by queries, so graphs drop to empty instead of flat-lining.

up == 0    # target is down
absent(up) # target vanished entirely

Many-to-many matching

Joining two vectors with different label sets raises many-to-many matching not allowed. Pin the join with on()/ignoring() plus group_left()/group_right().

a * on (job) group_left b   # ok
a * b                       # error

Aggregation drops the name

sum(), avg(), and the rest strip the __name__ label, so the result is nameless. Record the expression or aggregate without(__name__) when you need a stable name.

sum by (job) (rate(x[5m]))  # no metric name
job:http_requests:rate5m    # recorded name

⚠: Sum after rate, never before. sum(rate(http_requests_total[5m])) is correct. rate(sum(http_requests_total)[5m]) merges reset boundaries and yields a meaningless rate.