Grafana
Connect Grafana so Grafana alerts open incidents in Ace, and Ace investigates them with your alert rules, dashboards and datasources.
Ace reads Grafana alerts, rules, annotations and dashboards during investigations and queries your datasources, and Grafana-managed alerts open incidents.
| Category | Dashboards |
| Ace reads | Firing alerts, alert rules, annotations, dashboards, datasource queries and health |
| Ace can change | Annotations when an incident opens and resolves, and silences on the paging alert rule while an approved fix rolls out |
| Opens incidents | Yes, from Grafana alerts |
| Setup | Settings form |
Ace works with Grafana Cloud, Grafana Enterprise and self-hosted open-source Grafana. A self-hosted Grafana must use HTTPS and be reachable from the public internet, because Ace doesn't connect to private or internal addresses.
What Ace reads
- Active Grafana-managed alerts and alert rule definitions.
- Annotations around the incident.
- Dashboard search results, and a dashboard's panels and their queries.
- One PromQL, LogQL, TraceQL or SQL query at a time against a datasource, over the incident window.
- Datasource health.
After a fix, Ace checks whether the incident's alert is still active. It also discovers your datasources and the backend each one reads, for the System Map.
What Ace can change
| Action | What it does | Approval | Undo |
|---|---|---|---|
| Annotation | When an incident opens or resolves, Ace adds an annotation such as "Incident opened: <headline>", tagged ace. It goes on the alert's dashboard when the alert links to this Grafana. Otherwise it's org-wide, with the workload's tag | Runs on its own | Yes. Undo from the incident deletes the annotation |
| Silence | When an approved fix starts rolling out, Ace silences each alert rule that paged, for a bounded window. The silence matches only that rule, shows "Ace" as its creator and ends on its own | Runs on its own | Yes. Undo from the incident expires the silence |
Ace makes these updates only after an admin turns on Allow Ace to update this tool during incidents for the connection. See Actions and approvals.
On Grafana Cloud and Enterprise, Ace reads the service account's permissions and shows which updates it can make. Open-source Grafana may not report them, so there a missing permission shows up only when Grafana refuses an update.
What opens an incident
Grafana alerts reach Ace through a webhook contact point as they fire. Ace doesn't poll Grafana, so nothing arrives until the webhook is set up.
- A firing alert opens an incident or joins the one already open on the workload, and a resolved alert resolves it.
- Severity comes from the
severitylabel, which can becriticalorpage,highorerror,warningormedium, andinfoorlow. An alert without the label is high. - The route Ace creates skips alerts labeled
ace=ignoreand alerts with severityinfoornone. If you route alerts to Ace yourself, your notification policy decides what reaches Ace.
An alert opens an incident only when Grafana is routed to a workload. To route it, open the workload's settings, turn on Integration alerts under Investigation triggers and select Grafana under Alert sources.
Before you start
- The admin role in Ace. See Roles and permissions.
- The org Admin role in Grafana, to create a service account.
- On open-source Grafana, the service account needs the Admin role to read alert rules, because Viewer and Editor can't.
Credentials and permissions
| Credential | Where to create it | Permissions | Why Ace needs it |
|---|---|---|---|
| Service account token (required) | Administration > Users and access > Service accounts | Viewer role and alert.provisioning:read | Reads alerts, dashboards, annotations, datasources and alert rules |
| Same token (only for alerts) | alert.provisioning:write | Adds Ace's contact point and notification route | |
| Same token (only for actions) | annotations:create and annotations:delete, plus alert.silences:create and alert.silences:write (the fixed:alerting.instances:writer role, or Editor) | Adds and deletes annotations, and creates and expires silences |
Ace uses one service account token. Grafana has no read-only provisioning role, so on Cloud and Enterprise, fixed:alerting.provisioning:writer grants alert.provisioning:read along with the write access that alert wiring uses. Don't use the secrets reader role, which exports decrypted secrets.
Set up
Create the service account token
- In Grafana, open Administration > Users and access > Service accounts > Add service account (
https://<your-grafana>/org/serviceaccounts/create). - Set Display name to Ace and Role to Viewer, then choose Create.
- On Grafana Cloud or Enterprise, add the fixed role
fixed:alerting.provisioning:writer, andfixed:annotations:writerfor incident annotations. On open-source Grafana, give the account the Admin role instead. - For silences, add
fixed:alerting.instances:writer, or use the Editor role. - Choose Add service account token and Set expiration date, then Generate token and copy the token.
Connect Grafana in Ace
- In Ace, go to Settings > Data sources > Grafana.
- Fill in the form:
- Grafana URL: your Grafana's HTTPS URL, for example
https://example.grafana.net. - Service account token: the token you copied.
- Grafana URL: your Grafana's HTTPS URL, for example
- Choose Connect Grafana. If any validation check fails, Ace doesn't save the token.
Wire alerts
If the token has alert.provisioning:write, Ace creates a webhook contact point and a catch-all notification route, both named ace-<id>. The route continues to your other routes, so your existing routing is unchanged, and it also covers alert rules you add later.
Both stay editable in Grafana, and the Alerts section shows what Ace set up. Ace checks it again every hour and removes it when you disconnect.
To set it up yourself, choose Show manual setup in the Alerts section:
- Copy the webhook URL Ace shows.
- Choose Generate token and copy the token. Ace shows it once.
- In Grafana, add a contact point with the Webhook integration and paste the URL.
- Under Optional Webhook settings, leave Authentication Header Scheme at Bearer and paste the token into Authentication Header Credentials.
- In a notification policy, route the alerts that should open incidents to this contact point.
Allow updates (optional)
On the connected Grafana page, turn on Allow Ace to update this tool during incidents.
Confirm it's working
- The connection shows Connected after Ace reads your dashboards and active alerts.
- The Alerts section shows Waiting for first event until the first delivery Ace accepts, then Verified.
Disconnect
- In Ace, go to Settings > Data sources > Grafana and choose Disconnect. Ace removes its notification route and contact point, and if it can't, it tells you so you can remove
ace-<id>in Grafana yourself. - In Grafana, delete the service account or its token.
Next steps
- Actions and approvals: how observability updates and undo work.
- How Ace works: how alerts become incidents.
- Integrations: connect more data sources.
Prometheus
Connect Amazon Managed Service for Prometheus or an in-cluster Prometheus on Amazon EKS so Ace can query metrics during investigations and open incidents from Alertmanager alerts.
New Relic
Connect New Relic so alert issues open incidents in Ace, and Ace investigates them with your APM, OpenTelemetry, trace, error and NRQL data.