How Ace works
What happens from the first signal to a closed incident, and how to read what Ace shows you on the incident page and in Slack.
Ace opens an incident when an alert fires on a workload and investigates it across the sources you connected. It proposes or takes a fix under your action policy, checks that the incident's own alerts recover, and then closes the incident. This page walks through each stage and how to read the results.
What opens an incident
A workload is a service or job you care about, such as an API, checkout or a background worker. Only alerts routed to a workload open incidents on it, and you choose them under Which alerts start an investigation? when you create the workload.
| Source | What it is |
|---|---|
| Alerts from your tools | Monitors, alarms and issues from integrations such as Datadog, CloudWatch, Prometheus Alertmanager, Sentry or Grafana, routed to the workload. See Integrations. |
| Ace HTTP checks | Ace calls an endpoint on a schedule and opens an incident on an error rate spike, a latency regression or chronic slowness. See Proactive detection. |
| Metric watches | Ace learns each watched series' normal range and opens an incident when a deviation persists. |
| Early warnings | A forecast that a series will reach a known limit, or a capacity risk when it runs close to one. |
| A person | Run investigation now on a workload, or @Ace investigate <workload> in Slack. Both look at the last hour. |
Change-only sources, such as deploys and feature flag or config changes, go on the workload's timeline. Ace weighs them in every investigation, but they never open an incident on their own. Duplicate alert deliveries are kept once.
Severity
An incident is critical, high, warning or info. For alerts from your tools, Ace maps the alert's severity label like this:
| Alert label | Incident severity |
|---|---|
critical, page | Critical |
high, error | High |
warning, warn, medium, moderate | Warning |
info, none, low | Info |
| Missing or unrecognized | High |
If an alert with a higher severity joins an incident, the incident's severity goes up.
Grouping: one incident per workload
While a workload has an open incident, new alerts on it join that incident, so an alert storm becomes one incident. Each alert is listed under Signals with an occurrence count, and the incident resolves when none of them is still firing.
Alerts that join an incident don't each post to Slack. Instead the card updates when severity goes up, so a storm doesn't flood your channel. Manual investigations and forecasts always stay separate incidents.
Linked incidents
When one problem hits several workloads, such as a slow database behind several services, Ace opens one incident per workload and links a later incident to the first when they share an alert, or when the later one's cause sits in the first one's blast radius on the System Map.
The later incident shows Part of the first, and the first shows Also affecting, on the incident page and in Slack. The incidents stay separate, each with its own alerts, diagnosis and close. While the first incident's fix is in progress, the linked incident holds back its own fix so the two don't race.
The investigation
When Ace investigates
Ace investigates high and critical incidents on its own. Lower-severity incidents are recorded and watched, and Ace investigates them if they escalate or the alert is strong, such as an error rate spike on an HTTP check. It investigates again when something material changes, for example when severity goes up, the incident grows or a fix doesn't work.
You can also start an investigation at any time:
- Run investigation now or Investigate on a workload
- Investigate again on an incident, with optional context such as "it's not the deploy, check the queue"
@Ace investigate <workload>in Slack
What Ace reads
Each investigation starts from the incident's first alert and reads:
- the sources chosen under Which sources does Ace read? for the workload, plus any source that routes its alerts, with a source that has nothing relevant recorded as such
- deploys and flag, config and infrastructure changes before the incident, from every connected source that reports changes
- the System Map, with the workload's services and their dependencies
- the repositories linked to the workload
- memory: similar past incidents, the workload's fix history and recently rejected recommendations
- notes on the incident and the context you gave with Investigate again
- your operating rules, which shape the diagnosis and recommendations
What an investigation produces
| Part | What it is |
|---|---|
| Root cause | One hypothesis, or "inconclusive" when the evidence doesn't settle it. |
| Confidence | High, medium or low. |
| Category | Code defect, Deploy regression, Config or flag change, Infrastructure change, Capacity saturation, Upstream dependency, Data or state, or Cause not established. |
| Supporting evidence | The findings, per source, that back the root cause. |
| Contradicting evidence | Evidence that cuts against the causal path. |
| Other causes considered | Alternatives, each with the reason it was ruled out. |
| Open questions | What Ace couldn't establish. |
| Contributing factors | What made the incident worse or more likely. |
| How it connects | The path from change to effect to symptom, over System Map dependencies. |
| Recommended actions | Steps such as a scaling change, config tuning, a rate-limit change, traffic shaping, capacity planning, a code change or an infrastructure-as-code change. Each comes with a rationale, the evidence behind it and the alert that will show it worked. |
Reading the incident page
The incident page has four sections, Overview, Remediation, Evidence and Signals, and the bar at the top shows the time since detection at each stage: Detected, Investigating, Root cause, Remediation, Verifying, then Resolved or Closed.
The root cause card
The card shows:
- Root cause, or Most likely cause with Inconclusive when Ace couldn't settle it
- a three-bar meter with High confidence, Medium confidence or Low confidence
- whether the diagnosis is Preliminary or Final, plus Updating, Based on earlier evidence or Recorded before recovery if newer evidence has arrived. Check a preliminary diagnosis before you act on it, because it can still change.
- how many sources Ace checked, the category, the headline and what happened
- How it connects, with the change, effect and symptom each linked to its evidence
- Other causes Ace considered, with the reason each was ruled out
While an investigation runs, the card says "Correlating the signal with your workload, recent changes and connected sources." If the investigation fails, it says The investigation didn't finish and gives the reason.
Coding agents can read the full result, including contradicting evidence and open questions, with get_incident_context on the Ace MCP server.
What Ace is doing, and when it needs you
The What Ace is doing card shows Ace's goal, each attempt with its state (In progress, Waiting, Done, Will retry or Asked) and when Ace looks again.
When Ace needs a decision, the card becomes Ace needs you and offers the answers that apply: Review the step, I deployed it, I applied it, Investigate again or Close incident.
The side panel
The side panel shows status, severity, workload, what detected the incident, who started it, start time and duration, occurrences, people and links. Sources Ace read marks each source Findings, Nothing relevant or Skipped.
Notes you add appear under the incident, and Ace uses them in its next investigation after removing sensitive values. The Memory card shows what Ace brought from past incidents and what it learned from this one.
Slack updates
When Slack is connected, Ace posts each incident as a card and updates it in place:
- While the incident is open, the title shows severity and workload, and once it's resolved the title says the workload recovered.
- The buttons are I'm on it to acknowledge, Evidence and View suggested fix. Evidence opens the root-cause type, other causes considered, grouped alerts and the timeline.
- The footer has View workload and the incident reference, such as
INS-1018-ab12. - Linked incidents show Part of or Also affecting.
The RCA goes in the thread, titled RCA or Final RCA, or marked preliminary, with the confidence and hypothesis, key evidence, what happened, contributing factors and the recommended next step.
Ace asks for decisions with buttons in the incident's thread. If nobody answers, Ace reminds the thread, and on high and critical incidents it also mentions people who can answer.
Questions about early warnings stay in Ace and aren't posted. Ace asks only in Slack threads and on the incident page, and it doesn't page anyone through PagerDuty or incident.io, which are read-only integrations.
Talking to Ace in Slack
@Ace investigate <workload>shows a confirmation only you can see, with Start investigation and Cancel.- A reply in an incident's thread, such as "it's not the deploy, check the queue", makes Ace investigate again with your words as context.
- A request for an action, such as
@Ace scale checkout to 6 tasks, makes Ace build a plan from live state and post it for approval. See Actions and approvals.
Slack actions need your Slack user linked to your Ace account, so Ace can check your permissions.
Fixing the incident
Ace proposes a fix that matches the cause:
- undoing a correlated change, such as a deploy, a flag or config change, or an infrastructure change
- an operation on a running system for causes outside code, such as scaling a service or failing over
- a pull request for a code defect
Anything your action policy doesn't allow automatically becomes a question to a person.
After recovery, Ace can propose redoing work the incident lost, such as redriving an SQS dead-letter queue or replaying Trigger.dev runs. These steps wait for the fix to deploy, so the replayed work doesn't hit the same failure, and they always need approval. They never hold the incident open.
Recovery verification
Ace closes an incident only on evidence that the workload recovered:
- The change is live. A merged fix counts once it's deployed, which Ace confirms from your deploy provider or asks you to confirm with I deployed it.
- The incident's own alerts recover. The original alert or monitor resolves, fresh HTTP checks are healthy, or a measured series returns to its normal level and stays there.
- Ace keeps watching for the same failure after recovery.
Passing CI, a merged pull request or someone closing the incident never counts as recovery, because none of them show the workload is healthy. If the workload recovered before the change was applied, or after some other change, Ace records that it recovered on its own, and that result doesn't count toward earned autonomy. See Actions and approvals.
Use I applied it to tell Ace about a change you made yourself, with the time you made it. That time can't be in the future or more than a day before the incident began. Ace verifies recovery on the incident's alerts and never runs or undoes your change.
Closing and reopening
Ace closes an incident on its own when one of these happens:
- the alerts that opened it resolve
- its HTTP checks recover
- Ace verifies that an action recovered the workload
- for alerts with no "recovered" event, such as errors or failed jobs, the alert stays quiet after a deploy
An early warning closes when its condition clears. A metric-watch incident also closes when the watch stops, for example when someone mutes or deletes it, it runs out of data, or Ace learns the new level, but that close doesn't count as a recovery.
Closing an incident yourself
Select Close on the incident page and choose a reason under Why close?:
| Reason | Use it when |
|---|---|
| Incident resolved | It was a real incident and it's over. |
| Expected behavior | The alert reflects normal behavior. |
| Handled elsewhere | Your team deals with it somewhere else. |
| Not actionable | There's nothing to do about it. |
| Duplicate investigation | Another incident already covers it. |
Closing ends any running investigation, and Ace remembers your reason for that alert on that workload. See How memory works.
Reopening
If Ace closed the incident and the same alert fires within 24 hours, the incident reopens, and after 24 hours a new incident opens instead. If a person closed the incident, a new incident always opens.
What happens on close
On any close, Ace marks the firing alerts resolved, finalizes the diagnosis and withdraws proposed steps that no longer apply. It records what it learned in memory and refreshes the Slack card and the cards of linked incidents.
Next steps
Proactive detection
Actions and approvals
How memory works
Ace MCP server
How memory works
What Ace remembers from your incidents, when it learns, how memory shapes an investigation or quiets a known-noisy signal, how it fades, and how it stays inside your organization.
Proactive detection
How Ace watches your workloads on its own with HTTP checks, metric watches and early warnings, how it learns what normal looks like, and how to tune it.