Skip to content

docs: add Grafana dashboard import instructions to self-hosted monitoring page #312

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
seemeroland wants to merge 2 commits into
base: main
Choose a base branch
from
+61 −0
Open

docs: add Grafana dashboard import instructions to self-hosted monitoring page #312

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
aea7822
docs: add Grafana dashboard import instructions to monitoring page
seemeroland Jul 9, 2026
dd351d8
Merge branch 'main' into oz-agent/REMOTE-2110-grafana-dashboard
hongyi-chen Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions src/content/docs/platform/self-hosting/monitoring.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ description: >-
Prometheus, OTLP, or console to track worker health, task throughput,
and saturation.
---
import { Steps } from '@astrojs/starlight/components';

The `oz-agent-worker` daemon exports infrastructure-level metrics over [OpenTelemetry](https://opentelemetry.io/), giving your team real-time visibility into worker health, task throughput, and capacity. Combine these metrics with the [Oz dashboard](https://oz.warp.dev) for full observability across both the orchestration plane and your self-hosted compute.

Expand Down Expand Up @@ -148,6 +149,66 @@ All metrics use the `oz_worker_` prefix. Each worker process emits a distinct se

---

## Grafana dashboard

A ready-to-import Grafana dashboard is published in the [`grafana/` directory of the oz-agent-worker repository](https://github.com/warpdotdev/oz-agent-worker/tree/main/grafana). It covers all five areas of worker observability — health, throughput, latency, failures, and connectivity — and works with any Prometheus datasource.

### Dashboard sections

* **Worker Health** — Connected workers, active workers, tasks active, and fleet saturation gauge.
* **Task Throughput** — Task completion rate by result (succeeded/failed/cancelled), 5-minute success rate, and rejection rate.
* **Task Duration** — p50 / p95 / p99 latency over a sliding 5-minute window.
* **Failure Analysis** — Failure rate by reason (e.g., `image_pull`, `container_oom`, `unschedulable`) and by phase (assignment, backend, cleanup).
* **Connection Health** — WebSocket reconnect rate by reason, and a worker fleet info table showing each connected worker's ID, version, and backend.

The dashboard also includes an **Instance** variable so you can filter to a specific worker pod or see fleet-wide aggregations.

### Importing the dashboard

<Steps>

1. Enable Prometheus metrics on your workers. See [Enable Prometheus scrape](#enable-prometheus-scrape) above, or set `metrics.enabled=true` in the Helm chart.

2. In Grafana, go to **Dashboards** → **Import**.

3. Click **Upload JSON file** and select `oz-agent-worker-overview.json` from the [`grafana/` directory](https://github.com/warpdotdev/oz-agent-worker/tree/main/grafana), or paste the raw file URL:

```
https://raw.githubusercontent.com/warpdotdev/oz-agent-worker/main/grafana/oz-agent-worker-overview.json
```

4. When prompted, choose your Prometheus datasource and click **Import**.

</Steps>

You can also import via the Grafana API:

```bash
curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $GRAFANA_API_KEY" \

@oz-for-oss oz-for-oss Bot Jul 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 [SUGGESTION] [SECURITY] Expanding $GRAFANA_API_KEY in a command-line header can expose the bearer token in process listings; prefer a safer example such as a protected curl config or stdin-fed header.

--data-binary @oz-agent-worker-overview.json \
http://your-grafana/api/dashboards/import
Comment on lines +187 to +191

@oz-for-oss oz-for-oss Bot Jul 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [IMPORTANT] This posts the raw dashboard JSON to /api/dashboards/import, which Grafana does not accept as-is; document the required import payload schema or use /api/dashboards/db with a {dashboard, overwrite} wrapper.

```

### Prometheus alerting rules

The same `grafana/` directory includes [`alerts.yaml`](https://github.com/warpdotdev/oz-agent-worker/blob/main/grafana/alerts.yaml) — a set of Prometheus alerting rules covering the most common operational issues:

* **OzWorkerFleetEmpty** (critical) — No workers connected for 5 minutes.
* **OzWorkerDisconnected** (warning) — A specific worker has been disconnected for 2 minutes.
* **OzWorkerReconnectStorm** (warning) — A worker is reconnecting at more than 0.1/s.
* **OzWorkerFleetSaturated** (warning) — Fleet concurrency utilization is above 90%.
* **OzWorkerTasksRejected** (warning) — Workers are actively rejecting tasks.
* **OzWorkerHighFailureRate** (warning) — More than 10% of completed tasks failed.
* **OzWorkerImagePullFailures** (warning) — Container image pulls are failing.
* **OzWorkerContainerOOM** (warning) — Task containers are being OOM-killed.
* **OzWorkerUnschedulablePods** (warning) — Kubernetes task pods cannot be scheduled.

All thresholds are conservative starting points — tune them for your fleet size and SLOs. See [`grafana/README.md`](https://github.com/warpdotdev/oz-agent-worker/blob/main/grafana/README.md) for instructions on loading the rules into vanilla Prometheus or the Prometheus Operator.

---

## Sample PromQL queries

Direct mappings for common operational questions:
Expand Down
Loading