slop-stuff cheatsheets & experiments git

slop-stuff / Cloud, DevOps & observability

Grafana

Visualize everything.

Panels, variables, transforms, alerts, and provisioning as code.

observabilitypanelsalertsprovisioning

Grafana turns metrics, logs, and traces into dashboards you can read at a glance — then alerts on them. Query any data source, shape the result, and ship it.

Cheat sheet

The commands and values you reach for most. Everything else on this page expands on these.

TaskWhere / howKey syntax
Run GrafanaDocker, one containerdocker run -d -p 3000:3000 grafana/grafana
Add a data sourceConnections → Data sourcesprometheus · loki · postgres · tempo
Add a panelDashboard → Add → VisualizationExplore → Add to dashboard
Template variableDashboard → Settings → Variables$var · ${var:regex} · ${var:raw}
Reshape a resultPanel → Transform tabReduce, Filter by name, Join by field
AlertAlerting → Alert rulesReduce + threshold, route by label
Provision as codeYAML + env varsdatasources: · GF_SECURITY_ADMIN_USER
Time-aware PromQLBuilt-in macros$__rate_interval · $__interval · $__timeFilter()

v12: Current: Grafana 12. As of late 2025 the current major is Grafana 12 (11.6 was the last 11.x). New in 11/12: Explore Metrics & Explore Logs (query without PromQL/LogQL), Scenes-powered dynamic dashboards, the Trend and Canvas panels, a rebuilt table, and label-based access control (LBAC) for metrics.

Run & first dashboard

Five minutes from an empty server to a live panel: run it, sign in, wire a data source, add a panel.

1. Run

docker run -d -p 3000:3000 \
  grafana/grafana

2. Sign in

Open http://localhost:3000 and log in as admin / admin.

# change the password
# when prompted

3. Add a data source

Connections → Data sources → Prometheus, Loki, or Postgres.

http://prometheus:9090

4. New dashboard + panel

Dashboards → New → Add visualization → pick a query and panel type.

Dashboard → Add → Visualization

KEY: Queries live in panels. A dashboard is just a grid of panels; each panel runs its own query against a data source. Write the query in Explore first to iterate, then save it into a dashboard panel.

Persist data & plugins

Keep dashboards across restarts

docker run -d -p 3000:3000 \
  -v grafana-data:/var/lib/grafana \
  grafana/grafana

Install plugins at boot

GF_INSTALL_PLUGINS=grafana-clock-panel,\
grafana-piechart-panel

Data sources & queries

Every panel queries a data source through its own query language, then plots the result over a time range.

Prometheus

Metrics via PromQL. Add rate() to counters and use $__rate_interval.

rate(http_requests_total[$__rate_interval])
sum by (job) (rate(http_requests_total[$__rate_interval]))

Loki

Logs via LogQL. Filter with a stream selector, then pipe to parsers and aggregations.

{job="api"} |= "error"
sum by (level) (count_over_time({job="api"}[5m]))

PostgreSQL

Tables via SQL. $__timeFilter() applies the dashboard time range to a timestamp column.

SELECT time, value
FROM metrics
WHERE $__timeFilter(time)
ORDER BY 1
ShapeColumnsUse when
widetime, series A, series B, …One column per series (pivoted).
longtime, value, labelA value column plus a series-name column.

⌁: Time & units are panel-level. The dashboard time range picker applies to every panel. Set units in Panel → Standard options → Unit — common values: bytes, percent (0-100), percentunit (0.0-1.0), s, ms, short.

Iterate in Explore

Explore runs a query against one data source with the time range and query inspector visible. When a query looks right, use Add to dashboard to drop it into a panel. Grafana 11+ also ships Explore Metrics and Explore Logs for click-driven browsing without writing PromQL/LogQL.

# Explore → pick data source → write query
# → Add to dashboard → choose panel type

Panel types

Pick the visualization that matches the data shape, then tune thresholds, units, and the legend.

PanelWhat it showsUse for
Time seriesLines over timeMetrics, rates, latencies.
StatOne big numberCurrent value, e.g. uptime or errors.
GaugeValue on an arcUtilization against a max.
Bar gaugeHorizontal bars per seriesRanking several series at once.
TableTabular rows/columnsLogs, top-N lists, raw rows.
TextMarkdown / HTMLDocs, links, runbooks on a dashboard.

Thresholds

Color-code values by ranges. base is the default, green/red override it.

# Thresholds: 0 = green, 80 = base, 95 = red
# value 70 → green

Units

Format the raw value without changing the data.

Panel → Standard options → Unit
# bytes, percent, s, ms, short

Legend

Control series names and visibility.

# Legend format with labels:
{{job}} - {{instance}}
# or per-series overrides
More panel types

logs pie chart histogram state timeline status history geomap candlestick heatmap node graph flame graph trend canvas xy chart dashboard list alert list annotation list

Specialized panels for logs (Loki), distributions (histogram/heatmap), discrete state over time (state timeline/status history), and tracing (node graph, flame graph). New in Grafana 11/12: Trend (sparkline-style per-series stats), Canvas (free-form layout), XY chart (arbitrary x/y), and the dashboard / alert / annotation list panels.

Template variables

Make one dashboard reusable across hosts, jobs, and environments with $var placeholders.

TypeOptions come fromExample
queryA data source querylabel_values(up, job)
customA comma-separated listprod, staging, dev
intervalAuto time spans1m, 5m, 15m, 1h
constantA fixed hidden valueregion=us-east-1
datasourceThe data sources listpick Prometheus or Loki
textboxFree-text inputa hostname prefix

Multi-value

Enable Multi-value and Include All. The format changes interpolation: =~ regex for multi-select in PromQL.

# Multi-select as regex:
label=~"$jobs"
# Single: label="$job"

Chained (dependent)

Later variables read earlier ones: pick $region first, then $cluster narrows to it.

# $cluster query depends on $region:
label_values(kube_node_labels{region="$region"}, cluster)

!: Escaping. $var auto-escapes for the context; use ${var:raw} to disable escaping and ${var:regex} for a safe regex in label matchers. Chained variables refresh top-down when a parent changes.

Built-in variables & ad hoc filters
$__from          # range start (epoch ms)
$__to            # range end (epoch ms)
$__interval      # auto step, e.g. 15s
$__rate_interval # safe window for rate()

Ad hoc filters let viewers add key/value filters on the fly without editing the dashboard.

Transforms

Reshape query results in the panel before they render — no need to change the query itself.

  • Reduce — Aggregate rows into single values (sum, mean, min, max, last) per field.
  • Filter by name — Keep or remove fields matching a regex.
  • Merge — Combine several query results into one table by time.
  • Join by field — SQL-style join of two tables on a shared field value.
  • Group by — Bucket rows by field(s), then apply an aggregation.
  • Sort by — Order rows by a field ascending or descending.
  • Filter by value — Keep rows where a field passes a condition.
  • Rename by regex — Rewrite field/series names with capture groups.
  • Calculate field — Add a computed field from an expression.
  • Labels to fields — Turn label columns into data fields.

▦: Order matters. Transforms run top-to-bottom and each sees the previous one’s output. Put Filter by name before Reduce to slim the table before aggregating.

Organize & convert

Organize fields

# reorder, hide, or rename columns
# before the panel renders them

Convert field type

# string ↔ number ↔ boolean ↔ time
# fixes "no data" from string numbers

Alerts & notifications

Alert rules evaluate a query on a schedule, then route the result to contact points by label.

  • Normal — The condition is not met; the alert is quiet.
  • Pending — Condition met, but not yet for the full evaluation window.
  • Alerting — Condition has held; a notification is firing.
  • NoData — The query returned no data during the window.

Alert rule + condition

Create from a panel (Edit → Alert) or standalone (Alerting → Alert rules).

# Reduce + threshold expression:
when avg() of query(A, 5m, now)
IS ABOVE 0.9

Contact points

Where notifications go: email, Slack, PagerDuty, webhook, Telegram.

Alerting → Contact points → New
# pick an integration, fill secrets

Notification policies

The routing tree matches alerts to contact points by label.

# default policy → email
# severity=critical → pagerduty

Labels

Key/value pairs attached to an alert; they drive routing and silence matching.

severity: critical
team: api

↺: Routing is label-based. A notification policy matches on label = value. If an alert fires to the wrong place, check its labels against the routing tree before touching the contact point.

Evaluation timing

Rules evaluate on an interval (default 1m) over a for window. An alert stays Pending until the condition holds for the full for, then it becomes Alerting.

# Evaluate every 1m, fire after 5m:
# Interval: 1m · For: 5m

Provisioning as code

Declare data sources and dashboards as YAML, and override any setting with GF_-prefixed env vars.

Data sources

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true

Dashboards

apiVersion: 1
providers:
  - name: 'default'
    orgId: 1
    folder: ''
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards

Env vars

Every config key maps to GF_SECTION_KEY in the container.

GF_SECURITY_ADMIN_PASSWORD=secret
GF_SECURITY_ADMIN_USER=admin
GF_SERVER_HTTP_PORT=3000
GF_DATABASE_URL=postgres://…
GF_INSTALL_PLUGINS=grafana-clock-panel

Config file

Full config lives in grafana.ini; provisioned YAML is separate.

/etc/grafana/grafana.ini
# [server]
# http_port = 3000
# [security]
# admin_password = …

GF_: Env wins. Any grafana.ini value can be overridden by its GF_ env var at container start — no rebuild, no mounted config needed for most changes.

Secrets in datasources
datasources:
  - name: Postgres
    type: postgres
    url: db:5432
    jsonData:
      database: metrics
    secureJsonData:
      user: grafana
      password: secret

Pitfalls

Small behaviors that make a panel show nothing, wrong values, or the wrong series names.

Time range picker

Every panel is clipped to the dashboard time range. Use $__rate_interval in PromQL instead of a fixed [5m] so rates resize with the picker.

rate(requests_total[$__rate_interval])

Unit formatting

The unit changes the label, not the data. If values look wrong (e.g. bytes shown as a huge number), set the unit on the panel — not in the query.

Standard options → Unit → bytes

Template var escaping

$var is auto-escaped; a multi-value PromQL matcher needs the regex form or you’ll get a broken query.

label=~"${jobs:regex}"
# not: label="$jobs"

Series naming

Multiple series with identical names collapse to one line. Disambiguate with a Legend format or a rename transform.

Legend: {{job}} - {{instance}}

Data source access mode

access: proxy routes queries through the Grafana server; direct fetches from the browser. In Docker, or when the target isn’t reachable from the browser (CORS / private network), direct fails with no data.

access: proxy   # server-side, default
access: direct  # browser-side

No data & null gaps

Gaps appear where a series has no samples, not where it’s zero. For lines, enable Connect null values; to treat missing as zero, fill in the query with OR on() vector(0) (PromQL) or COALESCE(…, 0) (SQL).

# PromQL: fill missing with 0
requests_total OR on() vector(0)
# SQL: SELECT COALESCE(value, 0)

!: Data source auth. Provisioned data sources with credentials need secureJsonData (e.g. basicAuthPassword, token) — plain jsonData holds only non-secret settings. If a panel shows 401 / no data, check the source’s auth before the query.