observability

Grafana dashboards and alert rules for ScottyLabs services. Pulled into infrastructure/ as a flake input, materialized read-only into Grafana’s provisioning directory on infra-01.

Layout

dashboards/<service>/<name>.json
alerts/<service>/<name>.yaml

The folder under dashboards/ and alerts/ becomes the Grafana folder name. .forgejo/CODEOWNERS scopes each subdirectory to the team that owns the service and is regenerated by governance’s apply workflow whenever team membership or sentry flags change.

Authoring

Dashboards are authored in a local Grafana started by the devenv shell. The local Grafana runs anonymous with Admin role, points at the working tree, and reloads on file change. To ship a dashboard, build it in the UI, open Settings to JSON model, and copy the result into a file under dashboards/<service>/ before committing.

grafana-dev

Datasources are not provisioned locally; queries are validated against prod via tailscale once the dashboard ships. To preview against live data, set a viewer-scoped Grafana service-account token in GRAFANA_DEV_PROM_URL, GRAFANA_DEV_LOKI_URL, GRAFANA_DEV_TEMPO_URL.

Deploy

A push to main fires the Codeberg webhook at webhooks.scottylabs.org, which dispatches infrastructure/.forgejo/workflows/update-flake.yml and bumps flake.lock in infrastructure. Comin rebuilds infra-01 within a minute, after which Grafana’s provisioner picks up the new dashboards path and reloads.