# Ace AI > Ace is a team of AI agents built by Vratix, Inc. that investigates production incidents across infrastructure, telemetry and code, prepares fixes as reviewable pull requests, verifies recovery and remembers what it learned to prevent repeat failures. Investigation is read-only by default; changes follow the customer's approval policy. Key facts: - Agents: Telemetry, Infrastructure, Code and Recovery agents work the same case in parallel and share findings. - Integrations available: AWS, Kubernetes, GitHub, Datadog, Sentry, New Relic, Honeycomb, Prometheus, Grafana, Vercel, Cloudflare, Supabase, Slack, PagerDuty, incident.io, PostHog, LaunchDarkly, Elasticsearch, Trigger.dev, Inngest, PlanetScale, Axiom, LangSmith, Logfire, LiteLLM, CloudWatch, CloudTrail, AWS X-Ray. Coming next: Azure, MS Teams. - Triggers: CloudWatch alarms (polled or through Amazon SNS), Alertmanager alerts, Datadog monitors, New Relic workflows, Honeycomb trigger and SLO burn alerts, Sentry issues, Grafana and PostHog alerts, Cloudflare origin, edge and health alerts, Supabase project health, PagerDuty and incident.io incidents, failed Trigger.dev and Inngest runs, Ace HTTP health checks, or a request in the app or Slack. Only product reliability failures open incidents: CI runs, billing and budget alarms, feature flag changes and alerts marked ace:ignore are context. Signals from every source join the workload's open incident, so an alert storm opens one incident. - Actions: fix pull requests (with regression tests when reproducible, reviewed for correctness, data safety and operations, and one follow-up fix if Ace's own pull request fails CI), exact revert pull requests, rollbacks through an administrator-confirmed GitHub Actions workflow, Kubernetes restarts, scaling, patches and revision rollbacks, Vercel rollback and promotion, staged Trigger.dev replays, staged Inngest reruns and Inngest function pauses (always human-approved), staged PostHog flag rollouts, a LaunchDarkly flag kill switch, and PlanetScale deploy requests and backfills (always human-approved). Observability housekeeping runs automatically with Undo: Grafana and Axiom annotations, Honeycomb markers, Datadog downtimes and New Relic muting of the paging condition while a fix rolls out, and resolving or assigning the Sentry issue. Humans merge and the customer's pipeline deploys. Ace has no AWS write actions. - Autonomy: shadow mode by default, then per-plan approval. Production changes need an administrator or a granted approver. Automatic execution needs a confirmed workload scope, at least five verified recoveries with no worsened outcomes, a daily cap and a Slack cancellation countdown. Automatic merge of Ace's own pull requests is off by default and follows the same earned-autonomy rules. - Kubernetes: an outbound, namespace-scoped connector for EKS, GKE, AKS and on-premises clusters, installed with a pinned Helm chart from the open-source vratix-dev/ace-kubernetes-connector repository (Kubernetes 1.29–1.37). Reads are scoped per cluster and namespace; changes use a separate identity and the same approvals. - Memory: organization-wide search of past incidents by meaning, plus each workload's recent resolved incidents. - Collaboration: agents post in the incident's Slack thread as their own teammate; approvals and questions can be given by message. Slack replies are redacted before storage. PagerDuty and incident.io are read-only: Ace never writes to them. - MCP server: five read-only tools for coding agents (`get_incident_context`, `search_incidents`, `get_lessons`, `get_operating_rules`, `get_workload_context`), with organization-scoped, expiring, rate-limited tokens. See https://tryace.dev/docs/mcp.md. - Cost scans: scheduled scans of Trigger.dev and LiteLLM spend, unqueried Datadog custom metrics, and AWS Cost Optimization Hub or Compute Optimizer recommendations and Cost Explorer anomalies, with a weekly Slack digest. - Security: read-only investigation, least-privilege integration roles and tokens, signed or token-verified webhooks, log text scrubbed of literals, emails, IPs and credentials, separate permissions for changes. The AWS role denies secret values, decryption, sensitive data-plane reads and full SQL text. Ace is built and operated by Vratix, Inc., whose SOC 2 Type II report covers Ace (see https://trust.vratix.com). - Pricing: annual agreement covering an environment, investigation capacity and approved-change capacity. No per-seat or per-token billing. Quote-based; see https://tryace.dev/pricing.md. - In production with 7DIGIT, UNICEF and Warestack. - Contact: founders@vratix.com. Demo: https://tryace.dev/book-demo. Every page is available as Markdown at the same URL with `.md` appended (home: /index.md), or by requesting the page with `Accept: text/markdown`. The full site is in one file at https://tryace.dev/llms-full.txt. ## Product - [Overview](https://tryace.dev/index.md): What Ace is, how its agents work together, and how teams adopt it. - [Product](https://tryace.dev/product.md): System graph, parallel investigation, reviewed fixes, connector actions, recovery checks, system memory, earned autonomy, Slack and the Ace MCP server. - [Investigations](https://tryace.dev/investigations.md): How an investigation runs from alert to evidence-backed cause and proposed fix. - [Integrations](https://tryace.dev/integrations.md): Supported sources (AWS, Kubernetes, CloudWatch, X-Ray, Datadog, New Relic, Honeycomb, Prometheus, Elasticsearch, Sentry, Grafana, Axiom, Cloudflare, Supabase, GitHub, Vercel, Trigger.dev, Inngest, PostHog, LaunchDarkly, PlanetScale, Slack, PagerDuty, incident.io, LangSmith, Logfire, LiteLLM), what each reads and changes, and what is coming next. - [Security](https://tryace.dev/security.md): Read-only investigation, what each integration can reach, verified webhooks, separate authority to act and SOC 2 Type II. - [Pricing](https://tryace.dev/pricing.md): Annual agreement with coverage, investigation capacity and approved-change capacity. No per-seat or per-token billing. - [Book a demo](https://tryace.dev/book-demo.md): What a demo covers and how to book one. ## Field guides - [Field guides](https://tryace.dev/resources.md): Practical guides to evaluating incident agents, root cause analysis and reviewing agent changes. - [Audit your alerts in one afternoon](https://tryace.dev/resources/audit-your-alerts.md): A short method for finding the alerts that wake people up for nothing, the ones that are missing, and the pages worth keeping. - [A slow database, a waiting migration and the evidence between them](https://tryace.dev/resources/evidence-backed-root-cause-analysis.md): A worked PostgreSQL lock-queue investigation: follow blockers, test the explanation and check that the fix prevents a repeat. - [What to review before an agent changes production](https://tryace.dev/resources/governed-remediation.md): A concrete change proposal, approval boundaries and six failure cases to exercise before delegating production actions. ## Docs Developer documentation for engineers evaluating, using and integrating Ace. Start with [How memory works](https://tryace.dev/docs/concepts/memory.md) or the [Quickstart](https://tryace.dev/docs/quickstart.md). - [Ace documentation](https://tryace.dev/docs.md): Ace is an AI incident response team for your production systems. It investigates incidents across your infrastructure, telemetry and code, proposes fixes under your approval policy, verifies recovery and remembers what it learned. - [Quickstart](https://tryace.dev/docs/quickstart.md): Go from a new Ace workspace to your first investigation by connecting AWS, GitHub and Slack, adding a workload and inviting your team. - [Concepts: How memory works](https://tryace.dev/docs/concepts/memory.md): What Ace remembers from your incidents, when it learns, how memory shapes an investigation or quiets a known-noisy signal, how it fades, and how it stays inside your organization. - [Concepts: How Ace works](https://tryace.dev/docs/concepts/how-ace-works.md): What happens from the first signal to a closed incident, and how to read what Ace shows you on the incident page and in Slack. - [Concepts: Proactive detection](https://tryace.dev/docs/concepts/proactive-detection.md): How Ace watches your workloads on its own with HTTP checks, metric watches and early warnings, how it learns what normal looks like, and how to tune it. - [Concepts: Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals.md): Every change Ace can make, how your action policy decides what needs a person, who can approve, how undo works, how Ace verifies recovery, and how earned autonomy lets an action run automatically. - [Concepts: Operating rules](https://tryace.dev/docs/concepts/operating-rules.md): Write plain-language rules that tell Ace how your team wants incidents diagnosed and fixed, and understand exactly where Ace applies them and what they can't do. - [Integrations: Integrations](https://tryace.dev/docs/integrations.md): Every tool Ace connects to, what Ace reads and can change in each, which ones open incidents, and how connecting a tool works. - [Integrations: AWS](https://tryace.dev/docs/integrations/aws.md): Connect AWS so Ace can read CloudWatch, CloudTrail, X-Ray and your infrastructure during investigations, open incidents from CloudWatch alarms, scan for savings and, if you allow it, make approved changes. - [Integrations: Kubernetes](https://tryace.dev/docs/integrations/kubernetes.md): Connect a Kubernetes cluster so Ace can read resources, events and logs in the namespaces you choose, open incidents from failing pods and nodes and, if you allow it, make approved changes. - [Integrations: GitHub](https://tryace.dev/docs/integrations/github.md): Install the Ace GitHub App so Ace can read recent code changes, deploys and CI runs during investigations and, after approval, open fix and revert pull requests or roll back a deploy. - [Integrations: Slack](https://tryace.dev/docs/integrations/slack.md): Install the Ace Slack app so your team can start investigations with @Ace, follow incidents in your channels and approve Ace's actions in Slack. - [Integrations: Datadog](https://tryace.dev/docs/integrations/datadog.md): Connect Datadog so Ace can read metrics, logs, APM traces and monitors during investigations, open incidents from monitor alerts and mute the paging monitor while an approved fix rolls out. - [Integrations: Prometheus](https://tryace.dev/docs/integrations/prometheus.md): Connect Amazon Managed Service for Prometheus or an in-cluster Prometheus on Amazon EKS so Ace can query metrics during investigations and open incidents from Alertmanager alerts. - [Integrations: Grafana](https://tryace.dev/docs/integrations/grafana.md): Connect Grafana so Grafana alerts open incidents in Ace, and Ace investigates them with your alert rules, dashboards and datasources. - [Integrations: New Relic](https://tryace.dev/docs/integrations/new-relic.md): Connect New Relic so alert issues open incidents in Ace, and Ace investigates them with your APM, OpenTelemetry, trace, error and NRQL data. - [Integrations: Honeycomb](https://tryace.dev/docs/integrations/honeycomb.md): Connect Honeycomb so triggers and SLO burn alerts open incidents in Ace, and Ace investigates them by querying your traces and events. - [Integrations: Elasticsearch](https://tryace.dev/docs/integrations/elasticsearch.md): Connect Elasticsearch so Ace can search your logs, metrics and traces during investigations. - [Integrations: Axiom](https://tryace.dev/docs/integrations/axiom.md): Connect Axiom so monitor alerts open incidents in Ace, and Ace investigates them by querying your logs and events with APL. - [Integrations: Logfire](https://tryace.dev/docs/integrations/logfire.md): Connect Pydantic Logfire so Ace can query your application logs, traces, exceptions and metrics during investigations. - [Integrations: Sentry](https://tryace.dev/docs/integrations/sentry.md): Connect Sentry so issue and metric alerts open incidents in Ace, and Ace reads errors, releases, spans, logs and replays while it investigates. - [Integrations: LangSmith](https://tryace.dev/docs/integrations/langsmith.md): Connect LangSmith so Ace can read your AI workflow runs and traces during investigations. - [Integrations: LiteLLM](https://tryace.dev/docs/integrations/litellm.md): Connect your LiteLLM proxy so Ace reads model errors and spend during investigations, finds wasted model spend, and proposes fallback and budget changes for approval. - [Integrations: Vercel](https://tryace.dev/docs/integrations/vercel.md): Connect Vercel so your deploys join the change ledger next to incidents, and Ace can propose rollbacks, promotions, redeploys and firewall changes. - [Integrations: Cloudflare](https://tryace.dev/docs/integrations/cloudflare.md): Connect Cloudflare so Ace tells edge errors from origin errors, records DNS, rule and Worker changes next to your deploys, opens incidents from Cloudflare notifications, and proposes edge fixes for approval. - [Integrations: Trigger.dev](https://tryace.dev/docs/integrations/trigger-dev.md): Connect Trigger.dev so failed production runs open incidents, deploys join the change ledger, and Ace can replay or cancel runs and pause queues after approval. - [Integrations: Inngest](https://tryace.dev/docs/integrations/inngest.md): Connect Inngest so failed production runs open incidents, app syncs join the change ledger, and Ace can rerun, cancel or pause after approval. - [Integrations: Supabase](https://tryace.dev/docs/integrations/supabase.md): Connect Supabase so unhealthy services, Supabase health checks, read-only mode and a full disk open incidents, and Ace can fix pool size, disk and read-only mode after approval. - [Integrations: PlanetScale](https://tryace.dev/docs/integrations/planetscale.md): Connect PlanetScale so Ace reads your schema, deploy requests, Query Insights and bounded row counts, and can propose schema changes and backfills that a person approves. - [Integrations: PostHog](https://tryace.dev/docs/integrations/posthog.md): Connect PostHog so insight alerts open incidents, Ace measures who is affected, and Ace can roll a feature flag out in stages or switch it off after approval. - [Integrations: LaunchDarkly](https://tryace.dev/docs/integrations/launchdarkly.md): Connect LaunchDarkly so flag changes join the change ledger next to deploys, and Ace can propose a kill switch, a staged rollout or a version restore. - [Integrations: PagerDuty](https://tryace.dev/docs/integrations/pagerduty.md): Connect PagerDuty so a page joins the Ace incident already open for the same workload, and Ace reads timelines, alerts, on-call and service context during investigations. - [Integrations: incident.io](https://tryace.dev/docs/integrations/incident-io.md): Connect incident.io so a declared incident joins the Ace incident already open for the same workload, and Ace reads updates, alerts, on-call and your service catalog during investigations. - [MCP server: Ace MCP server](https://tryace.dev/docs/mcp.md): Give coding agents, code reviewers and other MCP clients read-only access to Ace's incident context, lessons from past fixes, your operating rules and the current state of a workload. - [Security and administration: Security and data](https://tryace.dev/docs/security/security-and-data.md): How Ace accesses your systems, what data it reads and keeps, how credentials and personal data are protected, and how your data stays inside your organization. - [Security and administration: Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions.md): The two workspace roles in Ace, what admins and members can each do, how approval permissions work, and how people sign in and join. ## Optional - [Trust center](https://trust.vratix.com): SOC 2 Type II status, policies and subprocessors. - [Privacy policy](https://vratix.com/legal/privacy-policy) - [Terms of use](https://vratix.com/legal/terms-of-use) - [RSS feed](https://tryace.dev/feed.xml) Examples on this website, including the "orders stalled after a schema change" incident, are synthetic and are not customer outcomes. --- Source: https://tryace.dev # Ace AI: an outage shouldn't stop your whole team > Your engineers should be building great things, not fixing outages or running cloud ops. Ace is a team of AI agents that investigates production incidents across your infrastructure, telemetry and code, prepares the fix as a reviewable pull request and checks that service has recovered. Ace is built and operated by [Vratix, Inc.](https://vratix.com), whose SOC 2 Type II report covers Ace. ## At a glance - **What it does:** investigates incidents, connects evidence across tools, proposes and prepares fixes, verifies recovery and remembers what it learned. - **Who it is for:** engineering teams that run production services on AWS and want fewer engineer hours spent on incident work. - **Agents:** Telemetry, Infrastructure, Code and Recovery agents work the same case and share findings with each other. - **Starts from:** CloudWatch alarms, Alertmanager, Datadog, New Relic, Honeycomb, Sentry, Grafana, PostHog and Cloudflare alerts, Supabase project health, PagerDuty and incident.io incidents, failed Trigger.dev and Inngest runs, Ace's own HTTP health checks, or a request in the app or Slack. Only product reliability failures open an incident, and an alert storm becomes one incident. - **Connects to:** AWS, CloudWatch, AWS X-Ray, CloudTrail, Datadog, New Relic, Honeycomb, Prometheus, Elasticsearch, Sentry, Grafana, Axiom, Cloudflare, Supabase, GitHub, Vercel, Trigger.dev, Inngest, PostHog, LaunchDarkly, PlanetScale, Slack, PagerDuty, incident.io, LangSmith, Logfire and LiteLLM. Kubernetes is available through an outbound connector installed with Helm. Azure and Microsoft Teams are coming next. - **Can change:** fix and revert pull requests, rollbacks through a GitHub Actions workflow your administrator confirms, Kubernetes restarts, scaling and patches, Vercel rollbacks and promotions, staged Trigger.dev replays, Inngest reruns and PostHog flag rollouts, Inngest function pauses, a LaunchDarkly kill switch, and PlanetScale deploy requests and backfills. Annotations, markers and alert muting while a fix rolls out run with Undo. Your team merges and your pipeline deploys. - **Works with your coding agents:** a read-only MCP server gives coding agents and other MCP clients the full incident context. - **Access model:** read-only investigation by default. Changes need separate permissions and follow your team's approval policy. - **In production with:** 7DIGIT, UNICEF and Warestack (deployed via 7DIGIT). - **Getting started:** [book a demo](https://tryace.dev/book-demo) or [request a quote](https://tryace.dev/pricing). Pilots run on your own incidents with scope and pricing agreed up front. ## A whole team, on the same case - **Telemetry** follows the failing request and shares the trace that sends the others to the right service. - **Infrastructure** checks the resources behind the failure and the services that depend on them. - **Code** tests recent changes against the findings, then prepares the patch and a regression test. - **Recovery** rechecks the original failure and downstream impact, and reopens the work if recovery fails. Ace coordinates them on one case, with one evidence trail. ## Know what depends on the failing service Ace maps the resources and dependencies in your infrastructure. Its agents use that graph to follow a failure across services, even when the alert only names one. See which callers share the failing database, where work is backing up and what a proposed change could affect. ## Review the fix where you ship code Ace opens a pull request with the patch, the evidence behind it and a test for the failure. Three Ace reviewers check correctness, data safety and operations, and if the pull request fails your checks, Ace makes one follow-up fix. Your engineers review and merge in GitHub. - **Prevent the next occurrence.** Clearing a queue restores service. A lock timeout, bounded retries and a contention test address the failure that would bring it back. - **Verify the full recovery.** Ace checks the deployed revision, API latency and downstream backlog. If a symptom remains, the investigation stays open. - **Supported operational actions.** For example, restoring the last healthy revision of a service, rolling back a Vercel deployment or replaying failed Trigger.dev runs in stages, with preconditions that stop the action if the target has changed. Your permissions and deployment setup determine which actions are available. ## Stop doing the same investigation twice Ace remembers the cause, the fix and your engineers' feedback, and finds similar incidents across your organization by meaning. Its agents check that history against the current system, reuse what worked and prepare changes that prevent the failure from returning. ## Delegate more as Ace earns it Give Ace a defined part of the operational work. Start with investigations, then authorize specific actions once you have reviewed the results. Your team sets the scope, limits and approval rules. | Step | Boundary | | -------------------------- | ------------------ | | Investigate | Read-only access | | Prepare a change | Review required | | Execute an approved action | Within your policy | | Verify and remember | Evidence retained | ## Your tools, one investigation Ace reads from your cloud, observability, delivery and code tools, then posts updates and approval requests in the incident's Slack thread, where each agent appears as its own teammate. See [integrations](https://tryace.dev/integrations). ## Start with the incidents your team knows Test Ace against real incident work. Agree on the systems it can access, the cases it should handle and the engineer time you want to save. Scope and pricing are agreed before you start. See [how pricing works](https://tryace.dev/pricing). --- The incident shown on the website (orders stalled after a schema change) is an illustrative example, not a customer outcome. --- Source: https://tryace.dev/product # Ace product: a team of agents for your cloud operations > Ace's agents investigate incidents, prepare fixes for review, verify recovery and keep what they learn, working from one map of your infrastructure. Start with incidents. Ace is a team of cloud agents built by Vratix, Inc. Its agents work from one graph of your infrastructure, the telemetry your services emit and the code that runs on them. ## Map the system before investigating it Ace builds a graph of your connected resources and their dependencies. Its agents use the same map to find affected callers, shared databases and services that could fail next. - Resource and service relationships - Observed dependencies and potential blast radius - Missing source coverage made visible ## Investigate in parallel, share the findings Infrastructure, telemetry and code agents pursue different leads at the same time and write to one shared incident record. A database finding redirects the trace search. A trace points to the change that needs review. When new evidence arrives, the investigation picks it up instead of starting again. - Logs, metrics, traces and recent changes - One incident for an alert storm, not hundreds - Hypotheses checked against contradictory evidence - Open questions kept alongside the diagnosis ## Prepare the fix for review Ace prepares fix and revert pull requests with the incident context attached. A fix carries the patch and, when the failure reproduces, a regression test, and is reviewed for correctness, data safety and operations before it opens. If Ace's own pull request fails your checks, it makes one follow-up fix. Rollbacks, Kubernetes changes and connector actions such as a Vercel rollback go through the same plan and approval. Your policies govern execution and your engineers merge. - Exact target, diff and expected effect - Approval requirements set by your team - Proposals and execution results kept together ## Check that service has recovered Ace revisits the original symptoms and checks affected dependencies. If recovery fails or the incident returns, it can open a follow-up investigation with the previous action and evidence attached. Retry budgets keep that loop bounded. - Recovery checks tied to the diagnosis - Remaining symptoms and regressions surfaced - Incomplete outcomes kept open for follow-up ## Keep the lesson, prevent the repeat Findings, engineer feedback and action outcomes become system memory. Ace searches past incidents across your organization by meaning, not just by name, to recognize familiar failures and avoid repeating unsuccessful work. It prepares lasting fixes and regression checks for the cause behind the incident. - Similar past incidents found across your organization - Working fixes and failed approaches remembered - Known causes carried into the next investigation ## Let results earn more autonomy Begin with read-only investigations and reviewed proposals. Production changes need an approver your administrator names. Delegate supported actions by target, with approval rules and execution limits. Ace keeps the diagnosis, authorized change and observed result together, so your team can decide what to delegate next. - Read-only investigation first - Named approvers for production changes - A record of decisions, changes and outcomes ## Work where your team already works Each agent posts in the incident's Slack thread as its own teammate. Approve a plan with a message, ask about it, and follow each step as it runs. Coding agents can pull the full incident context through Ace's MCP server, and scheduled scans flag wasted spend in connected sources. - Slack updates, questions and approvals in one thread - Read-only MCP server for incident context and search - Cost findings from AWS, Datadog, Trigger.dev and LiteLLM ## How an investigation starts - A CloudWatch alarm, read directly from your AWS accounts, or an Alertmanager alert or Amazon SNS notification delivered to Ace by webhook (setup steps for each source are in the [integration docs](https://tryace.dev/docs/integrations)). - An alert or webhook from a connected source: Datadog monitors, New Relic workflows, Honeycomb trigger and SLO burn alerts, new or regressed Sentry issues, Grafana and PostHog alerts, Cloudflare origin, edge, health check and tunnel alerts, and failed production Trigger.dev and Inngest runs. - A page: a triggered or reopened PagerDuty incident, or a live incident.io incident. A page for a failure another source already reported joins that incident. - Supabase project health: unhealthy services, failed or paused projects, read-only mode, health advisor findings and disk above 90%. - An Ace HTTP health check that fails against a URL configured on a workload. - A request from the Ace app or Slack. Only product reliability failures open an incident. CI runs, billing, budget and spend alarms, low-urgency pages, muted issues and anything marked `ace:ignore` are kept as context instead. Deploys, feature flag changes, DNS and edge configuration changes and schema migrations from connected sources join the incident timeline, so a flag flip or a DNS change is weighed against the latest deploy. Missed notifications are caught by polling. Signals from every source join the affected workload's open incident, so an alert storm becomes one investigation instead of hundreds. The incident closes only when all of its signals are quiet. ## What Ace can change today | Action | What happens | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Fix pull request | Ace prepares a minimal code or infrastructure-as-code repair in an isolated sandbox, adds a regression test when the failure is reproducible, reviews it for correctness, data safety and operations, and opens a pull request with the incident context. | | Revert pull request | An exact reversal of a change, opened as a pull request. | | Rollback | Redeploys a known-healthy revision through a GitHub Actions workflow an administrator has confirmed. | | Kubernetes change | Restart, scale, patch configuration or roll back to a captured revision in namespaces you grant to a separate writer identity. | | Vercel rollback or promotion | Instant rollback to a previous production deployment, or promotion of a deployment. | | Trigger.dev replay | Replays failed runs in stages, for example 10 first and then the rest, each run at most once. | | Inngest rerun or pause | Reruns failed function runs in stages once the fix is live, each run at most once. Pausing a function and resuming it each need a person's approval, and each undoes the other. | | PostHog flag rollout | Rolls a feature flag out in stages, for example 10%, 50% and 100%, with signal checks between stages. | | PlanetScale change | Schema changes through deploy requests (and their revert), and bounded backfills. These always need a person's approval and never run automatically. | | LaunchDarkly kill switch | Turns off the flag behind an incident, frozen to the flag version Ace saw. Refused if the flag has changed since. Needs a person's approval or an explicit policy, and turning it back on is a separate approved change. | | Observability housekeeping | Grafana and Axiom annotations and Honeycomb markers for the incident and fix, a Datadog downtime or New Relic muting rule on the paging condition while an approved fix rolls out, and resolving or assigning the Sentry issue. These run automatically unless your policy says otherwise, and each has Undo. | If Ace's own pull request fails your required checks, it makes one follow-up fix without weakening tests or checks. Your engineers merge pull requests and your pipeline deploys them, unless you turn on automatic merge for a service where Ace has earned it. Ace does not run `terraform apply`, has no AWS write actions, and does not change live infrastructure outside these actions. Every connector action needs an administrator's grant first. ## How autonomy is earned New organizations start in shadow mode: Ace records what it would do without doing it. Teams can then require approval for each plan, by its exact content, in the app or Slack. Automatic execution is a separate policy for one confirmed workload and destination. It needs at least five verified recoveries and no worsened outcomes in that scope, stays within a daily cap, and announces each run in Slack with a cancellation countdown before it writes anything. Production changes need an approver: an organization administrator, or a member an administrator has granted that action type and environment. When someone else tries to approve in Slack, Ace mentions the approvers in the thread. Automatic merge is off by default. Where it is on, Ace merges only its own earned-autonomy pull requests, after required checks and every Ace reviewer pass, and after a cancellation countdown in Slack. ## Incident tools and paging PagerDuty and incident.io pages can open an investigation, and on-call, escalation policies, owning teams and service catalogs inform it. Responders' notes are read as hypotheses, not as evidence. Ace never writes to PagerDuty or incident.io: it adds no notes or timeline items and never resolves a page. Findings and the resolution go to the incident's Slack thread. ## System map Edge proxies such as Cloudflare zones sit in front of the Vercel and AWS origins they serve. APM, error tracking and paging services (New Relic, Honeycomb, Datadog, Sentry, PagerDuty, incident.io) link to the infrastructure service of the same name that they watch, Supabase projects resolve from a workload's database URL, and LaunchDarkly flags link to the repositories that reference them. ## Kubernetes An outbound connector for EKS, GKE, AKS and on-premises clusters, installed with a pinned Helm chart (or a manifest) from the open-source [ace-kubernetes-connector](https://github.com/vratix-dev/ace-kubernetes-connector) repository. It supports Kubernetes 1.29 to 1.37 and works with Argo CD or Flux. Your cluster API is never exposed. Each cluster has its own identity and namespace scope, and Secret values never enter evidence. Agents read resources, events, logs, rollouts, services, network policies and node conditions. Changes such as restarting, scaling or patching a workload use a separate identity you grant, and go through the same plan, approval and audit flow as every other action. Automatic repair is narrower still: expiring, reversible namespace patches to values you approve in advance. ## Example investigation (illustrative) Orders stalled after a schema change. A reporting transaction held a relation lock, a migration's DDL queued behind it, and application reads and outbox delivery queued behind the DDL. Ace's agents connected `pg_locks`, a request trace and the deployment that retried lock acquisition without a timeout. The proposed recovery paused retries and cancelled only the waiting DDL, with approval required. The prevention pull request added a lock timeout, bounded retries and a contention regression test. Recovery checks confirmed latency and outbox delivery returned to normal. This example is synthetic; see the [worked write-up](https://tryace.dev/resources/evidence-backed-root-cause-analysis). ## Related - [Investigations](https://tryace.dev/investigations) - [Integrations](https://tryace.dev/integrations) - [Security](https://tryace.dev/security) - [Pricing](https://tryace.dev/pricing) --- Source: https://tryace.dev/investigations # Ace investigations: start with the alert, get a cause you can check > Infrastructure, telemetry and code agents work in parallel and share what they find. Your engineers get the evidence, the open questions and a proposed fix without running every search themselves. ## How an Ace investigation works 1. **Start from the alert.** Ace opens a case from the failing signal and the service it names. Further alerts for the same workload, from any source, join that case instead of opening new ones. 2. **Split the work.** Telemetry follows the failing request, Infrastructure checks the resources and dependents behind it, and Code tests recent changes against the findings. 3. **Share findings.** Agents write to one shared incident record, and a finding from one redirects the others. For example, a database lock finding sends the trace search to the session that holds it. When significant new evidence arrives, the investigation continues. 4. **Test the explanation.** Hypotheses are checked against contradictory evidence. Unresolved questions stay visible next to the diagnosis. 5. **Propose the fix.** Ace prepares a pull request or a supported action with the exact target, the evidence and recovery checks. 6. **Verify and remember.** Ace rechecks the original symptoms, keeps the case open if they remain and retains the lesson, so similar incidents anywhere in your organization start from it. ## Understand the fix before you approve it Trace the patch back to the failure it addresses. Review the target, the tests and the recovery checks before a change reaches production. ## Before you connect Ace **Does Ace need write access to investigate?** Investigation starts with scoped, read-only access to connected sources. Actions have separate permissions and review requirements. **Can engineers inspect the evidence?** Findings retain their source context, so engineers can examine the evidence behind the diagnosis and proposed action. **How do we decide what Ace can do?** Your team selects the actions, target systems and approval requirements. Start with read-only investigations and expand that scope after reviewing the results. ## Further reading - [A slow database, a waiting migration and the evidence between them](https://tryace.dev/resources/evidence-backed-root-cause-analysis): a worked root cause analysis. - [Audit your alerts in one afternoon](https://tryace.dev/resources/audit-your-alerts): find the noisy pages and the missing alerts. --- Source: https://tryace.dev/integrations # Ace integrations: connect your tools, connect the evidence > Give Ace's agents access to the tools your engineers already use. They follow the failure across infrastructure, telemetry and code, and share findings in Slack. ## Infrastructure - **AWS**: Maps your resources and dependencies, and checks config changes, health events and cost. (Topology · config history · health · cost) - **Kubernetes**: Shows what is running, failing or newly rolled out, and can restart, scale or patch workloads. (Resources · events · logs · rollouts) - **Cloudflare**: Shows whether errors come from the edge or your origin, and spots DNS and config changes. (HTTP analytics · Workers · audit logs) - **Supabase**: Database health, load and logs to explain slow or failing requests. Unhealthy projects start an investigation. (Health · metrics · logs · advisors) - **PlanetScale**: Schema and branch history to explain database failures. Ace proposes schema fixes as deploy requests. (Schema · branches · deploy requests) - **CloudTrail**: Who changed what in AWS, and when, checked against the incident timeline. (Change events · actors) ## Code & delivery - **GitHub**: Connects an incident to the code, deploys and CI runs behind it, and opens fix pull requests. (Repositories · deployments · CI runs · pull requests) - **Vercel**: Ties a failure to the deploy behind it, and can roll back or promote a deployment. (Deployments · promotions) - **PostHog**: Shows which users an incident affects, and rolls feature flags out in stages. (Product impact · feature flags) - **LaunchDarkly**: Flag changes are weighed against deploys as a possible cause. Ace can propose a kill switch. (Flag changes · evaluations · kill switch) - **Trigger.dev**: Failed background jobs start an investigation, and Ace can replay them in stages. (Runs · deploys · replays) - **Inngest**: Failed function runs start an investigation, and app syncs show what changed. Ace can rerun failed runs in stages or pause a function. (Runs · step traces · syncs · reruns) ## Observability - **Datadog**: Traces, metrics and errors to follow a failing request. Monitor alerts start an investigation. (APM · error tracking · monitors · downtimes) - **Sentry**: Errors and releases to pin down what broke. New or regressed issues start an investigation. (Issues · events · releases) - **New Relic**: Traces, golden signals and errors to follow a failure. Alerts start an investigation. (NRQL · issues · traces · deployments) - **Honeycomb**: Queries and traces to find what changed in a failing request. Trigger and SLO alerts start an investigation. (Queries · traces · SLOs · markers) - **Prometheus**: Metrics to measure impact and confirm recovery. Alertmanager alerts start an investigation. (Time series · workload signals) - **Grafana**: Alerts start an investigation, and dashboards get a note when a fix ships. (Alerts · alert rules · annotations) - **Elasticsearch**: Logs from the incident window to find the first error. (Logs · error search) - **Axiom**: Logs and events from the datasets you choose, searched over the incident window. (Logs · events · annotations) - **LangSmith**: LLM traces to explain failures in AI features. (LLM traces · application context) - **Logfire**: Application traces and logs to follow a failure. (Application telemetry) - **LiteLLM**: Gateway errors and model spend, to explain AI failures and wasted cost. (Gateway health · model errors · spend) - **CloudWatch**: Metrics, logs and alarms to measure impact. Alarms start an investigation. (Metrics · logs · alarms) - **AWS X-Ray**: Request traces to follow a failure across services. (Traces · service relationships) ## Collaboration - **Slack**: Where Ace posts updates, asks questions and gets approvals, in the incident thread. (Updates · approvals · action progress) - **PagerDuty**: Pages start an investigation, and on-call and service details show who owns the failure. (Pages · on-call · change events) - **incident.io**: Incidents start an investigation, and the service catalog and on-call show who owns the failure. (Incidents · catalog · on-call) ## Coming next - **Azure**: Azure infrastructure and service dependencies. - **MS Teams**: Incident updates in Microsoft Teams. ## Access Each connection has its own permissions and scope, and investigation is read-only. See [security](https://tryace.dev/security.md) for what each connection can reach, and [product](https://tryace.dev/product.md) for what opens an investigation and what Ace can change. We check source coverage and required permissions with your team. Missing a tool? Email [founders@vratix.com](mailto:founders@vratix.com) with what you use and what Ace needs to read. --- Source: https://tryace.dev/security # Ace security: read access to investigate, your permission to act > Ace investigates with read-only access. You choose the sources it can reach. Preparing or executing a change requires separate permissions and the policies your team sets. ## Principles - **Read-only by default.** Investigation uses read access only. You choose which sources Ace can reach, and reading an incident never grants write access. - **Least privilege.** Each integration has its own scoped permissions, and each key lists the read scopes Ace needs separately from the scopes for alert wiring and actions. Secret values stay out of AWS and Kubernetes evidence, and log text is scrubbed of literals, emails, IPs and credentials. - **Approval before action.** Every change is a plan you approve in full. Automatic execution is a separate, earned policy. Every decision and result is recorded. ## What each integration can reach - **AWS:** read-only role. Resource configuration and relationships, Config history, Health events, CloudWatch, X-Ray and CloudTrail data, plus cost recommendations and anomalies. Secret values, decryption and sensitive data-plane access are explicitly denied, and Performance Insights returns tokenized SQL and wait events, never full SQL text. Ace has no AWS write actions. - **Kubernetes:** outbound connector, so your cluster API is never exposed. Per-cluster identity and namespace scope. Changes use a separate identity you grant. - **GitHub:** the Ace GitHub App on the repositories you choose. Opening a pull request needs write access there. Your team merges. - **Datadog:** metrics, logs, monitors, APM and Error Tracking with read scopes. Alert wiring, downtimes and usage cost reads are separate scopes you can leave out. - **Prometheus:** query permissions scoped to the Amazon Managed Prometheus workspace you connect. - **Slack:** investigation updates and approval requests in channels you choose. Replies in Ace threads are redacted for secrets, card numbers, bank and phone numbers before they are stored. - **Supabase:** a fine-grained access token with read permissions only, for the projects you list. Classic tokens are refused, requests are limited to read endpoints before they leave Ace, and Ace never runs SQL. - **Cloudflare:** a custom API token with read permissions on one account and the zones you choose. Workers Observability has no read-only level in Cloudflare, and Notifications Edit is needed only if Ace wires its own alerts. - **LaunchDarkly:** a Reader token for flag changes and status, never flag values or targeted keys. A separate writer token is optional, and is used only for a kill switch that a person or your policy approved. - **PagerDuty:** A read-only key covers pages, on-call and services. A full-access key is needed only if Ace adds its own webhook subscription, and Ace never writes to PagerDuty. Signed webhooks are verified. - **incident.io:** View scopes cover incidents, the service catalog and on-call. Ace never writes to incident.io, and private incidents are ignored. - **New Relic:** a user key for a dedicated user with a custom role. NRQL runs only in the accounts you allow and only over the incident window. Muting needs a separate permission. - **Honeycomb:** queries only the datasets you allow, no further back than seven days. The configuration key needs Manage permissions on queries, SLOs, triggers and markers. - **Inngest:** a key with Read permission for one environment, optionally limited to the apps you list. A separate Write key is optional, and is used only for reruns and pauses a person or your policy approved. Signing keys are refused. - **Other connectors:** Sentry, Grafana, Axiom, LiteLLM, Vercel, Trigger.dev, PostHog and PlanetScale use credentials you provide, with optional project or dataset allowlists. PlanetScale reads use a read-only password. - **Webhooks and writes:** provider signatures or tokens are checked in constant time, and connector credentials are stored encrypted. Every write needs an administrator's grant first. - **Ace MCP server:** read-only incident context and search for coding agents. Tokens are scoped to your organization, expire after 90 days by default, are rate-limited, and every call is logged. ## Changes follow your approval policy 1. **Shadow mode first.** New organizations start in shadow mode: Ace records what it would do without doing it. 2. **Approve the exact plan.** Approval is bound to the displayed plan. A changed plan needs approval again. 3. **Production needs an approver.** Production changes need an administrator or a member an administrator has granted that action and environment. Ace mentions the approvers in Slack when someone else tries. 4. **Automatic execution is earned.** It applies to one confirmed workload and destination, after verified recoveries with no worsened outcomes. Daily caps apply and each run is announced in Slack with a cancellation countdown. 5. **A record of everything.** Decisions, policy changes, executions and recovery results are kept in an append-only audit trail. ## Data in an investigation Resource metadata, logs, metrics, traces and the code and deployments you connect. Findings and action outcomes are kept as the investigation record. During your security review we confirm data handling and retention for your deployment. ## Compliance - **Company:** Ace is built and operated by Vratix, Inc. - **SOC 2 Type II:** covers Vratix, Inc., including Ace. The report is available through the trust center. - **GDPR:** listed as compliant in the trust center. - **Subprocessors:** Anthropic and OpenAI (AI models), AWS and Google Cloud (infrastructure), GitHub and others. Nine in total, listed in the trust center. - **Policies:** 25 policies and 35 monitored controls at [trust.vratix.com](https://trust.vratix.com), including access control, authentication, encryption, data handling and remote access. A security questionnaire is available on request. ## Report a security concern Email [founders@vratix.com](mailto:founders@vratix.com) with a description and steps to reproduce. Please leave credentials and customer data out of the first message, and we will coordinate sensitive details with you. --- Source: https://tryace.dev/pricing # Ace pricing: one annual agreement, priced on the work Ace takes on > Ace is sold as an annual agreement scoped to your production environment and the work you delegate to it: investigations and approved changes. Every quote names the environment and quantities it covers. There is no per-seat, per-token or per-agent-minute billing. ## How pricing works One order form, three parts: | Part | What you get | What sets the quote | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | Coverage | Ace maintains context for a named production environment: connected accounts, services and repositories, with agreed support. | The inventory in scope, how often it is processed, retention and support needs. | | Investigation capacity | An annual allowance of distinct investigations that end in an evidence-backed finding or an agreed rule-out. | Your deduplicated incident volume across the teams in the rollout. | | Approved-change capacity | An annual allowance of authorized changes, delivered as reviewed pull requests that meet agreed acceptance checks. | The change work Ace can take on, the repositories involved and who approves it. | Capacity is pooled across the teams covered by the agreement. ## What you are not billed for - **Seats.** Everyone on the covered teams can use Ace. - **Supported integrations within your scope.** Connecting another supported source does not add a fee on its own. - **Model usage.** Tokens, tool calls, agent run time and retries are Ace's cost to manage, not a line on your invoice. - **Repeat alerts for the same case.** Alerts and retries tied to one incident count as one investigation. - **Safety.** Read-only defaults, approval controls and the investigation record are part of every agreement. ## Start with a pilot Most teams start with a bounded pilot on their own incidents. Before it starts we agree on: - the systems and repositories Ace can access, - representative incidents, including repeat failures, - how engineer time is measured, including time spent checking and correcting Ace's work, - success criteria and a decision date. Scope and pricing are agreed before you start. ## What we need for a quote 1. The services, regions and repositories in your first rollout. 2. Roughly how many distinct incidents you handled in the last 30 to 90 days. 3. The kinds of changes you would want Ace to prepare. 4. Your security, identity and data-handling requirements. ## How a quote is put together 1. **Scope first.** We agree the environment, teams and repositories in the first rollout, and what is out of scope. 2. **Measured work, not headcount.** Investigation and change allowances come from your own recent incident and change history, not from company size or the number of engineers. 3. **Requirements that change delivery.** Security, identity, retention and support requirements are confirmed before the proposal, because they change what we commit to. 4. **Checked against your alternative.** The proposal should leave your team clearly ahead of what incident work costs you today, measured in engineer time after reviewing and correcting Ace's work. You receive a written proposal that names the scope and quantities. Pricing is agreed before any rollout. ## Pricing questions **How is Ace priced?** As one annual agreement with three parts: coverage for your environment, investigation capacity and approved-change capacity. The quote names the quantities included. **Do you charge per seat or per token?** No. Seats, supported integrations within scope and model usage are not billed separately. **What counts as an investigation?** One distinct incident case that ends in an evidence-backed finding or an agreed rule-out. Repeated alerts and retries for the same case count once. **What counts as an approved change?** One authorized change objective delivered as a reviewed pull request that meets the acceptance checks agreed for it. Follow-up commits for the same objective are included. **Can we start small?** Yes. A focused deployment covers one environment and a smaller allowance. Expand to more teams and capacity as results justify it. **How do I get a quote?** Fill in the form at [tryace.dev/pricing](https://tryace.dev/pricing), email [founders@vratix.com](mailto:founders@vratix.com) or [book a demo](https://tryace.dev/book-demo). --- Source: https://tryace.dev/book-demo # Book an Ace demo: see Ace work through an incident > See the agents investigate together, inspect the evidence and review a proposed fix. We'll discuss how that workflow fits your stack and on-call team. ## What we'll cover - Your systems and on-call workflow - Agents, evidence and a proposed fix - Access, approvals and a pilot for your team No preparation required. A rough outline of your infrastructure helps us make the session relevant. ## Book - Calendar: [cal.com/ivan-ivanov/demo](https://cal.com/ivan-ivanov/demo) - Email: [founders@vratix.com](mailto:founders@vratix.com) - Pricing and quotes: [tryace.dev/pricing](https://tryace.dev/pricing) --- Source: https://tryace.dev/resources # Ace field guides for the on-call team > Practical ways to test a diagnosis, review a fix and measure the incident work your team can delegate. ## [Audit your alerts in one afternoon](https://tryace.dev/resources/audit-your-alerts) Operations · 6 min read · Updated October 19, 2026 A short method for finding the alerts that wake people up for nothing, the ones that are missing, and the pages worth keeping. Markdown: https://tryace.dev/resources/audit-your-alerts.md ## [A slow database, a waiting migration and the evidence between them](https://tryace.dev/resources/evidence-backed-root-cause-analysis) Investigations · 7 min read · Updated October 17, 2026 A worked PostgreSQL lock-queue investigation: follow blockers, test the explanation and check that the fix prevents a repeat. Markdown: https://tryace.dev/resources/evidence-backed-root-cause-analysis.md ## [What to review before an agent changes production](https://tryace.dev/resources/governed-remediation) Operations · 7 min read · Updated October 17, 2026 A concrete change proposal, approval boundaries and six failure cases to exercise before delegating production actions. Markdown: https://tryace.dev/resources/governed-remediation.md RSS: https://tryace.dev/feed.xml --- Source: https://tryace.dev/resources/audit-your-alerts --- title: "Audit your alerts in one afternoon" description: "A short method for finding the alerts that wake people up for nothing, the ones that are missing, and the pages worth keeping." category: Operations author: The Ace team published: 2026-10-19 updated: 2026-10-19 canonical: https://tryace.dev/resources/audit-your-alerts --- # Audit your alerts in one afternoon > A short method for finding the alerts that wake people up for nothing, the ones that are missing, and the pages worth keeping. Most teams know their on-call rotation has too much noise. Few have looked at it as data. An afternoon with the last 30 days of pages is enough to cut the noise by a third and to find the two or three alerts that should exist and do not. You need an export of every page or notification from the last month, with the alert name, the time it fired, who acknowledged it and how long it stayed open. Most paging tools can produce this as a spreadsheet. ## Count before you judge Group the export by alert name and sort by count. The top of that list is almost always a surprise. A handful of alerts usually account for more than half of all pages, and a few of them have not led to an action in weeks. Write down three numbers for each of the top twenty alerts: - how many times it fired, - how many times someone did something because of it, - how often it resolved itself before anyone looked. An alert that fires often and resolves itself is asking for a longer evaluation window or a different threshold. An alert that fires often and gets acknowledged without action is asking to be deleted or turned into a ticket. ## Ask what the page tells a person to do For each alert you keep, answer one question: if this woke you at 3am, what would you do first? If the answer is "look at a dashboard", the alert is a symptom and the page should link to the dashboard and name the question to ask there. A useful page has four parts: 1. what is failing, in terms a customer would notice, 2. since when, 3. the first place to look, 4. who owns the service. A template that fits in a notification: ```text Checkout errors above 2% for 5 minutes Since: 14:32 UTC First look: checkout service dashboard, error breakdown by endpoint Owner: payments team, escalate to platform after 15 minutes ``` Alerts that describe a cause, such as high CPU, tend to page people for things that do not matter. Alerts that describe an effect, such as checkout errors above two percent for five minutes, tend to page people for things that do. ## Find what is missing Now read the last five real incidents. For each one, note how it was detected. If a customer or a colleague noticed first, there is a gap. Write the alert that would have caught it and what threshold would have fired early enough to matter, without firing every day. Be honest about the second half. A new alert that pages twice a week becomes the next entry in the noisy list. ## Decide with a simple rule Put every alert into one of four groups. | Group | What it means | What to do | | --- | --- | --- | | Page | Needs a human now and has a clear first step | Keep, and add the four parts above | | Ticket | Needs a human this week | Route to a queue, not to a phone | | Dashboard | Useful context, never urgent | Remove the alert and keep the graph | | Delete | Has not led to action in a month | Delete it and write down why | The aim is not fewer alerts for their own sake. It is that every page earns the interruption. ## Make it stick Schedule the same review for the first week of every month and keep it short. Track two numbers over time: pages per on-call shift and the share of pages that led to an action. If the first falls and the second rises, the rotation is getting healthier. Add one more habit. After every incident, ask in the review whether an alert fired, whether it was the right one, and whether it fired early enough. Each answer either changes an alert or confirms one, and the set improves with every incident instead of only growing. When pages carry a clear first step and the evidence is one click away, whoever is on call, human or automated, starts from the right place. [Ace](/product) investigates from the same alerts, so a cleaner page set also gives it a cleaner starting point. --- Source: https://tryace.dev/resources/evidence-backed-root-cause-analysis --- title: "A slow database, a waiting migration and the evidence between them" description: "A worked PostgreSQL lock-queue investigation: follow blockers, test the explanation and check that the fix prevents a repeat." category: Investigations author: The Ace team published: 2026-10-15 updated: 2026-10-17 canonical: https://tryace.dev/resources/evidence-backed-root-cause-analysis --- # A slow database, a waiting migration and the evidence between them > A worked PostgreSQL lock-queue investigation: follow blockers, test the explanation and check that the fix prevents a repeat. “The database is slow after a deployment” describes a sequence of events. It does not explain the failure. Before approving a fix, an engineer needs to know what is waiting, what it is waiting on and why the proposed change will release it. Consider this illustrative PostgreSQL incident: order requests reach 8.4 seconds at p99, the outbox stops delivering events and database CPU remains at 19%. A schema migration started just before the alert. These are invented example observations, not customer results. ## Separate the symptom from the mechanism The slow request spends almost all its time in a database span. That narrows the investigation, but a long span can include several kinds of waiting. Check connection acquisition, query execution and lock waits separately. In this case, the waiting reads point to migration session 2317. That migration is itself waiting for reporting session 1284, which has an open transaction and a table lock. The migration requests an `AccessExclusiveLock`. Later reads can queue behind it even though it has not acquired the lock yet. PostgreSQL calls this a soft block: a conflicting waiter ahead in the queue can block a later request. [PostgreSQL documents both hard and soft blockers](https://www.postgresql.org/docs/current/functions-info.html#FUNCTIONS-INFO-SESSION). The chain to establish is: ```text Reporting transaction holds a table lock → schema migration waits for an exclusive lock → later application reads wait behind the migration → order requests and outbox delivery stall ``` Low CPU is consistent with this explanation, but it does not prove it. The blocking relationships do the important work. ## Capture the queue before changing it Start with a known affected backend PID from your database tooling. The following read-only query shows that session’s immediate blockers. Replace the example PID, run it against the relevant database and use a role authorized to inspect activity. Some session fields are restricted without additional monitoring privileges. See the [PostgreSQL activity-view documentation](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW). ```sql SELECT a.pid, a.application_name, a.state, a.wait_event_type, a.wait_event, now() - a.xact_start AS transaction_age, pg_blocking_pids(a.pid) AS blocked_by FROM pg_stat_activity AS a WHERE a.datname = current_database() AND a.pid = 2317; ``` Follow the returned blocker PIDs in a second observation. Record when each sample was taken. This is a changing system, so separate queries are not an atomic snapshot of the entire lock chain. Avoid turning the diagnostic into a tight polling loop; `pg_blocking_pids` briefly accesses shared lock-manager state. Use `pg_locks` and the activity details to confirm the relation and requested lock modes. An old transaction alone is insufficient. You need the relationship between that transaction, the queued migration and the affected reads. The [locking documentation](https://www.postgresql.org/docs/current/explicit-locking.html) describes which modes conflict. ## Make each finding change the next check The infrastructure investigation identifies the blocking chain. That should redirect telemetry toward the other callers of the same table. In this example, the outbox publisher’s database waits match the migration session, explaining why order delivery stopped alongside the API. The code investigation then checks the migration runner. Does it bound lock acquisition? What happens after a timeout or cancellation? If it retries indefinitely, cancelling one waiting statement may only remove the queue until the runner tries again. A concise evidence record would contain: - **Observed:** affected reads wait behind session 2317, which waits behind session 1284. - **Observed:** the outbox reads the same relation and stops advancing during that window. - **Observed:** the deployed runner retries the migration without a bounded lock wait. - **Inferred:** removing the queued DDL and preventing immediate retries should allow the affected reads to proceed. - **Still to check:** whether another lock or downstream failure will remain after that queue clears. Each observation needs a source and a time. Keep the inference distinguishable from what was directly measured. ## Write down what would disprove the diagnosis If affected requests still wait after the queued migration is gone, this explanation is incomplete. If the outbox has no matching database wait, investigate its own failure path. A nearby deployment that does not touch the affected path is weak evidence by itself. These are useful checks before and after an action. They prevent a plausible narrative from becoming the only explanation the team considers. ## Verify recovery and test the permanent fix Review the recovery plan with the owner of the migration. Recheck the session identity and state before any intervention. A PID copied from an earlier screenshot is not a durable action target. This article is a diagnostic walkthrough, not a blanket instruction to terminate sessions. For prevention, test a local lock timeout and bounded retry policy under the same contention. PostgreSQL’s `lock_timeout` limits waiting to acquire locks. It is not an overall query-runtime limit. [The configuration reference explains the distinction](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-LOCK-TIMEOUT). A useful regression holds the reporting transaction open, starts the migration and attempts an application read. Verify that the runner exits or retries within the intended bounds and does not repeatedly stall reads. Then check that the migration succeeds when contention is removed. Recovery evidence should include the original request latency, the lock queue and outbox progress. Save the failed condition and the regression with the incident so the next schema change starts with a check your team has already learned to need. --- Source: https://tryace.dev/resources/governed-remediation --- title: "What to review before an agent changes production" description: "A concrete change proposal, approval boundaries and six failure cases to exercise before delegating production actions." category: Operations author: The Ace team published: 2026-10-15 updated: 2026-10-17 canonical: https://tryace.dev/resources/governed-remediation --- # What to review before an agent changes production > A concrete change proposal, approval boundaries and six failure cases to exercise before delegating production actions. An automated fix should be easier to review than a command pasted into an incident channel. The reviewer needs the exact target, the change, the evidence behind it and the conditions that will stop execution. Here is a practical review format for teams introducing automated production changes. The examples describe engineering controls to evaluate. They are not a promise that every action type is supported by Ace or your deployment. ## Bind approval to a specific change “Roll back the orders service” leaves too much open. A useful proposal identifies the environment, deployment, current revision and intended replacement. For a pull request, it identifies the repository, base commit and patch the reviewer saw. Use a record like this: ```text Incident: ACE-842 Reason: deployment 842 introduced unbounded migration retries Target: orders-platform / production / migration runner Expected current revision: Proposed change: Evidence: Checks before execution: Approval: Stop if: Recovery checks: Fallback: ``` The placeholders must be resolved before execution. A service label is useful for people but is not a substitute for the provider’s resource identity. Approval should become invalid when a material part of the plan changes. If another deployment lands while the reviewer is reading, refresh the proposal. Do the same when the patch, workflow inputs or target scope changes. Quietly carrying an old approval onto a new plan defeats the review. ## Separate the permissions for each step Reading a repository, opening a pull request, merging it and deploying it are distinct capabilities. Grant only the capabilities the workflow needs. For GitHub, configure branch protection or rulesets for required reviews and checks, and inspect bypass permissions. Required reviews are useful only if the automation cannot quietly bypass them. GitHub also provides settings for dismissing stale approvals and requiring approval of the most recent reviewable push. [Review the protected-branch controls](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches) against your actual workflow. For deployment workflows, bind the action to a configured target and an allowed workflow. Reject an ambiguous service-to-deployment mapping. A system that cannot establish the target should leave a proposal for an engineer to resolve. ## Decide what recovery means before running the change Take an orders incident with a backed-up outbox. A healthy API response does not establish that pending events are being delivered. The review needs both checks. Write the observation plan in terms the on-call engineer can verify: - **Original symptom:** request error rate and latency for the affected endpoint and workload. - **Shared dependency:** the lock or connection wait that supported the diagnosis. - **Downstream progress:** outbox age and delivery rate, including whether the backlog is draining. - **New damage:** errors in dependent services and failures introduced by the change. - **Observation window:** enough traffic and time to judge the result, chosen for this system. Avoid a universal five-minute “healthy” rule. A low-traffic service or a periodic worker may need a different window. If the required telemetry is absent, record recovery as unverified. ## Handle the response you never received Suppose a provider accepts an action, but the connection drops before the client receives the response. Retrying immediately can create a second change. Declaring failure may also be wrong: the first action could still be running. Keep an operation identifier and reconcile against provider state before deciding to retry. Use the provider’s idempotency mechanism where available. When execution cannot be established, show an unknown outcome and a next check. The [AWS Builders’ Library on idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) explains the underlying retry problem. The action history should distinguish these states: proposed, approved, dispatched, accepted, completed and recovery verified. A rejected request, a failed workflow and an unknown result need their own records. Do not compress them into one green or red badge. ## Exercise the boundaries in a sandbox Before granting production execution, run cases that test the controls themselves: 1. Change the target revision after approval. Execution should stop or require renewed review. 2. Request a resource outside the allowed scope. It should remain untouched. 3. Reject or expire an approval. No provider operation should be dispatched. 4. Lose the provider response. Confirm that reconciliation does not duplicate the action. 5. Let the action succeed while the symptom persists. Recovery must remain unresolved. 6. Make the fallback unsafe because the system has changed. It should require a fresh decision rather than an automatic reversal. Retain the provider-side result as well as the application’s action record. That is how another engineer can distinguish “the UI says blocked” from “nothing reached production.” ## Leave a fix that survives the incident Immediate recovery and prevention deserve separate review. Cancelling a queued migration may restore requests, while bounding retries and testing lock contention address why the same queue could return. Link the follow-up patch, regression and owner to the incident. Later, verify that the patch shipped and the test exercises the failure. Memory is useful when it changes the next decision, not when it merely makes an old incident easier to search. Ace starts with read-only investigations and separately authorized actions. [Review the access model](/security) and use these checks to decide which work your team is ready to delegate. --- Source: https://tryace.dev/docs # Ace documentation > Ace is an AI incident response team for your production systems. It investigates incidents across your infrastructure, telemetry and code, proposes fixes under your approval policy, verifies recovery and remembers what it learned. Source: https://tryace.dev/docs Ace is an AI incident response team. When something breaks in production, Ace opens an incident, investigates it across the tools you connect and explains the root cause with the evidence for and against it. It then proposes a fix, such as a pull request, a rollback or a Kubernetes restart, which runs under your approval policy. After the fix ships, Ace checks that the original alert recovered and remembers what worked for next time. ## How it fits together 1. An alert from your monitoring tools, an Ace HTTP check, a metric watch or an early warning opens an incident on the affected [workload](https://tryace.dev/docs/concepts/how-ace-works), so an alert storm on one workload becomes a single incident. 2. The investigation reads the metrics, logs, traces, cloud resources, recent deploys and code connected to that workload, and pulls similar past incidents from [memory](https://tryace.dev/docs/concepts/memory). 3. You get a root cause in the incident's Slack thread and on the incident page, with a confidence level, the evidence for and against it, and what's still unknown. 4. Ace proposes a fix as a pull request, a rollback or an operation, and by default a person approves each change. Changes run automatically only where Ace has [earned autonomy](https://tryace.dev/docs/concepts/actions-and-approvals#earned-autonomy) under your action policy, and your [operating rules](https://tryace.dev/docs/concepts/operating-rules) set how your team wants the work done. 5. Once the fix is deployed, verification watches the original alert until it's healthy. If the problem comes back, the incident reopens. Investigation is read-only. Ace changes a system only if you gave it a credential that can write there and an admin turned on actions for it, and even then code and infrastructure changes need approval unless your policy lets Ace act on its own. Updates that only change what your team sees in a monitoring tool, such as an annotation or a downtime on the monitor that paged, run on their own once you allow them. Each one can be undone. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). ## Where to start - [Quickstart](https://tryace.dev/docs/quickstart): From a new workspace to your first investigation. - [How memory works](https://tryace.dev/docs/concepts/memory): What Ace remembers, how it recalls it, and how it fades. - [Integrations](https://tryace.dev/docs/integrations): What Ace reads and can change in each tool, and how to connect it. - [Ace MCP server](https://tryace.dev/docs/mcp): Give coding agents read-only incident context. - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): What Ace can change, and who approves it. - [Security and data](https://tryace.dev/docs/security/security-and-data): What Ace reads and keeps, and how it's protected. ## If you're an AI agent, start here Read [`/llms.txt`](https://tryace.dev/llms.txt) first, since it lists every page in these docs with a one-line summary. To fetch a page as Markdown, add `.md` to its URL, for example [`/docs/concepts/memory.md`](https://tryace.dev/docs/concepts/memory.md), or send `Accept: text/markdown`. [`/llms-full.txt`](https://tryace.dev/llms-full.txt) has every page in one file. To read Ace incidents, lessons and operating rules while you work on code, connect to the [Ace MCP server](https://tryace.dev/docs/mcp). ## Get help Our SOC 2 Type II report, policies and subprocessors are in the [Trust Center](https://trust.vratix.com). For anything else, [book a call with our team](https://tryace.dev/book-demo). --- Source: https://tryace.dev/docs/quickstart # Quickstart > Go from a new Ace workspace to your first investigation by connecting AWS, GitHub and Slack, adding a workload and inviting your team. Source: https://tryace.dev/docs/quickstart Guided setup takes about ten minutes and has five steps: **Workspace**, **Connect**, **Workloads**, **Team** and **Activate**. Investigations start once our team activates your workspace, and you can set up everything else before then. ## Before you start You need: - An invite. Ace is invite-only, so you can create an account only if a teammate invited your email address or our team gave you access. Otherwise, [book a call with our team](https://tryace.dev/book-demo). - A Google or GitHub account for that email address. - For AWS, someone who can launch a CloudFormation stack in your account. Connecting an AWS Organization also needs StackSets trusted access, which an admin of the management account turns on. See [AWS](https://tryace.dev/docs/integrations/aws). - For GitHub, someone who can install GitHub Apps on your organization. See [GitHub](https://tryace.dev/docs/integrations/github). - For Slack, permission to install apps in your workspace. See [Slack](https://tryace.dev/docs/integrations/slack). Ace saves your progress on the workspace, so you can reload, switch tabs or hand over to another admin and pick up at the same step. ## 1. Sign in Go to [app.tryace.dev](https://app.tryace.dev) and choose **Continue with Google** or **Continue with GitHub**, using the email address you were invited with. If a teammate invited you to an existing workspace, Ace shows a **Join** button with the workspace name. Join it and your team's integrations and workloads are already there, so you can skip to [step 6](#6-run-your-first-investigation). To start fresh, choose **Create a separate workspace**, which declines the invite because each person belongs to one workspace. ## 2. Name your workspace Whoever creates the workspace becomes its first admin, and only admins see guided setup. 1. Enter a **Workspace name**. Ace suggests one from your email domain. 2. Check the **Timezone**, which Ace uses for weekly reports. 3. Select **Continue**. ## 3. Connect your stack The **Connect** step suggests AWS, GitHub and Slack first, and each one opens in a side panel. | Integration | What it gives Ace | How you connect it | | --- | --- | --- | | **AWS** | A map of your accounts, services and their dependencies, so Ace knows where to look. | Select **Connect** and follow the guided setup. It creates a read-only IAM role with CloudFormation, which you review and launch in your AWS console. | | **GitHub** | Your repositories and deploys, so Ace can tie an incident to the change behind it. | Select **Install app** and choose which repositories Ace can see. | | **Slack** | Where you talk to Ace, with findings and approvals landing in your channels. | Select **Add to Slack** and approve the install. | Access is read-only, so Ace can't change anything until an admin allows actions, and each change then follows your [approval policy](https://tryace.dev/docs/concepts/actions-and-approvals). AWS mapping runs in the background while you continue. Under **More sources** you can connect error tracking, metrics, logs, paging, edge, feature flags and databases, such as Kubernetes, Sentry, Datadog, Grafana and Prometheus. These add evidence to investigations, and their alerts can start them. See [Integrations](https://tryace.dev/docs/integrations). Select **Continue** once at least one integration is connected. If nothing is connected yet, select **Skip for now** and connect later from **Settings**. ## 4. Add a workload A workload is a service or job you care about, such as your API, checkout or a background job. Alerts, checks and investigations attach to a workload, and Ace investigates across everything it touches. In the **Workloads** step, select **Add your first workload**. The **New workload** flow asks four questions: 1. **What is it?** If AWS is connected, pick its services from your system map, then give it a **Name** such as `Orders API`. Ace reads the optional **Description** when it investigates. 2. **Which code builds it?** Link the GitHub repositories that build it, so Ace can read recent changes and deploys from them. 3. **Which alerts start an investigation?** Choose the alert sources whose alerts for this workload start an investigation. You can also add an **HTTP health check** that Ace calls on a schedule. Nothing is selected by default, and with nothing selected Ace investigates only when someone asks. 4. **Which sources does Ace read?** Choose the integrations Ace reads while it investigates this workload. Ace always reads a source that routes alerts to the workload, and you can change the rest later in the workload's settings. Check the summary, then select **Create workload**. Only admins can link repositories, choose alerts and choose sources, because those control what Ace reads and what starts an investigation. Members can create a workload and add HTTP checks. See [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions). ## 5. Invite your team and activate ### Invite your team Invite the teammates who will follow investigations and approve Ace's actions. In the **Team** step: 1. Enter one or more email addresses, separated by commas or spaces. 2. Choose a role: **Member** (the default) or **Admin**. 3. Select **Add**, then **Send**. Invites are valid for 7 days, and each person signs in with Google or GitHub using the invited email address. ### Choose Slack channels Once Slack is connected, an admin picks the channels Ace posts to in **Settings** > **Slack**. Add a channel to **Insights** for incidents and investigation starts, and to **Weekly reports** for summaries. To approve actions or start investigations from Slack, a person's Slack account must be linked to their Ace account, so Ace knows who is acting and can check their permissions. The person who installed Slack is linked automatically, and an admin links everyone else in **Settings** > **Team**. ### Activate your workspace A new workspace starts out pending activation. Your setup is saved and integrations are validated, but Ace doesn't investigate or take actions, so alerts that arrive are recorded as incidents without an investigation and Slack mentions get a reply that Ace isn't activated yet. The **Activate** step summarizes your setup and offers **Book a call**. On the 30-minute call you agree with our team on what Ace watches and who approves actions, and our team then activates your workspace. Select **Go to workspace** to finish. A getting-started checklist stays on your home page until you complete or dismiss it. ## 6. Run your first investigation Once your workspace is active, any member can start an investigation, and Ace looks at the workload over the last hour. **In Ace** - Right after you create a workload, select **Run investigation now**. - On any workload, select **Investigate**. You can add a note to steer the investigation before it starts. **In Slack** In a channel where Ace is a member, or in a direct message to Ace, write: ```text @Ace investigate orders api ``` Ace matches the name to a workload and replies with a confirmation only you can see, with **Start investigation** and **Cancel**. Your Slack account must be linked to your Ace account. Ace also investigates on its own. An alert routed to a workload starts an investigation when its severity is high or critical, and alerts with no severity count as high. A failing HTTP health check starts one when the failure is serious, such as an error-rate spike, while smaller failures are recorded and watched. Ace posts the incident and its findings to your **Insights** channel, and you can reply in the thread to direct it. To read the results, see [How Ace works](https://tryace.dev/docs/concepts/how-ace-works). ## Next steps - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): From first signal to closed incident, and how to read the results. - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): What Ace can change, and who approves it. - [Integrations](https://tryace.dev/docs/integrations): Connect more sources for evidence and alerts. - [Security and data](https://tryace.dev/docs/security/security-and-data): What Ace reads, what it keeps and how it's protected. --- Source: https://tryace.dev/docs/concepts/memory # How memory works > What Ace remembers from your incidents, when it learns, how memory shapes an investigation or quiets a known-noisy signal, how it fades, and how it stays inside your organization. Source: https://tryace.dev/docs/concepts/memory When a new incident opens, Ace recalls similar past incidents, what fixed them and what made things worse, and uses that to weigh the new evidence. Memory comes from your own incidents, your engineers' feedback and the verified results of Ace's actions, and it is never shared between organizations. Memory never grants permissions. Your [action policy](https://tryace.dev/docs/concepts/actions-and-approvals) sets what Ace may change, and your [operating rules](https://tryace.dev/docs/concepts/operating-rules) can narrow it further. ## What Ace remembers Ace keeps two kinds of memory. An incident memory summarizes one incident that reached an outcome, and a learned pattern records one observation about an alert or a fix. ### Incident memory An incident memory is a short summary for future investigations: | Part | What it holds | | --- | --- | | Headline and workload | The incident's headline and the workload it happened on. | | What happened | Ace's summary of the incident. | | Root cause | The root cause Ace established and its confidence, or a note that no cause was established. | | Actions | The actions taken in the incident, and how far each one got. | | Engineer notes | Recent notes your engineers added to the incident. | | Outcome | How the incident ended, including how verification judged Ace's actions. | Ace writes one only when the incident has a root cause or an action that made progress. It holds summaries and notes, and raw logs and metrics stay out. Ace searches incident memories by meaning, so a search for "duplicate summaries sent to customers" can find an incident titled "Replay re-sent completed jobs". Engineer notes and Slack replies are redacted before they're stored, which removes credentials, tokens, keys, signed URLs, email addresses, payment card numbers, phone numbers and other personal or secret details. See [Security and data](https://tryace.dev/docs/security/security-and-data#redaction). ### Learned patterns A learned pattern is a small observation, such as "this alert has been a false positive" or "this fix made things worse on this workload". Its confidence rises with supporting evidence and falls with contradicting evidence. An incident's **Memory** card lists the patterns Ace used (**Used**) and learned (**Learned**), each with its scope and its count of supporting and contradicting observations. | Pattern, as shown in Ace | What it records | Where it comes from | | --- | --- | --- | | Known false positive | An alert that turned out not to be a real problem. | An engineer closes an incident as **Expected behavior**, **Not actionable** or **Duplicate investigation**. An Ace HTTP check also records one when a single failure clears on its own. | | Suppressed pattern | An alert your team handles somewhere else. | An engineer closes an incident as **Handled elsewhere**. Ace's own checks also record one when they treat a failure as known noise. | | Confirmed incident | An alert that was a real incident. | An engineer closes an incident as **Incident resolved**. Ace also records one when an HTTP check failure turns out to be part of a real incident. | | Pattern that came back | An alert Ace had quieted that turned into a real incident. | Ace's own checks. Ace also weakens the pattern that quieted the alert. | | HTTP error signature | How a failing HTTP check failed: status code, error type or response. | Ace HTTP checks. | | Alert signature | An alert from one of your integrations, by provider and alert name. | Each alert that opens or joins an incident. | | Runbook preference | Recommendations your team rejected, and how each fix Ace applied turned out (recovered, did not recover, or made it worse). | Your decisions on recommendations, and recovery verification after actions. | | How your team responds | Replies, reactions and button clicks on Ace's Slack messages. | Slack. | To mark an alert as noise, close the incident as **Expected behavior** or **Not actionable**, or use **Handled elsewhere** if another team owns it. To confirm a real incident, close it as **Incident resolved**. Repeated closes can eventually quiet an alert that Ace raises itself. Alerts from your own tools always open an incident, so fix those monitors at the source. ## When Ace learns | Event | What Ace learns | | --- | --- | | An incident closes, for any reason | Ace writes or refreshes its incident memory. | | An engineer closes an incident in Ace and picks a reason | A pattern for that alert on that workload (see the table above). **Expected behavior** and **Not actionable** also weaken any earlier sign that the alert was a real incident. | | Ace verifies the result of an action | How that fix turned out on that workload. Ace also refreshes the incident memory with the outcome. | | An engineer rejects a recommendation | That your team doesn't want that recommendation on that workload. | | An alert opens or joins an incident | An alert signature. | | An Ace HTTP check fails and recovers | Whether the failure was a false positive, a real incident or known noise, and how it failed. | | An engineer replies, reacts or clicks in Slack | How your team responds to that alert. If the engineer edits or deletes a reply, the stored text follows. | Learning never blocks the work that produced it. If a write fails, the incident still closes and the action still completes, and Ace catches up later. ## How Ace recalls memory When Ace weighs the evidence and settles on a cause and recommendations, it pulls in: 1. similar incidents across your organization, found by meaning from the current headline, what happened and Ace's leading hypotheses 2. this workload's recent incidents, whether or not they read alike 3. what Ace's past actions achieved in those incidents: verified recovered, did not recover, made the incident worse, or recovered on its own 4. the workload's fix history, with every fix Ace applied and how often each one recovered the workload, failed or made things worse 5. recommendations recently rejected on the workload 6. learned patterns attached to the incident's alert If part of memory is unavailable, Ace continues with the rest, so recall never blocks an investigation. ## How memory shapes an investigation Memory changes what Ace considers and proposes, but never the evidence Ace collects. - A past cause or fix applies only when current evidence points to the same service, change or failure mode. A similar headline isn't enough. - A fix that made an incident worse on a workload is never proposed again on that workload, however it's worded. - A fix that didn't recover the workload is proposed again only when the evidence shows why this time is different. The rationale explains the difference, and the fix's history appears in the proposal's evidence for reviewers. - A fix that worked is a candidate only when the evidence points to the same cause. - If your team recently rejected a similar recommendation on the workload, Ace proposes it again only with new evidence. - Ace treats recalled text as untrusted data, like any other incident content. ## How memory quiets an alert Learned patterns can stop Ace from raising an alert that has repeatedly been noise, but only for alerts Ace raises itself: [Ace HTTP checks](https://tryace.dev/docs/concepts/proactive-detection), [metric watches](https://tryace.dev/docs/concepts/proactive-detection) and [early warnings](https://tryace.dev/docs/concepts/proactive-detection). Alerts from your own tools, such as a Datadog monitor, a CloudWatch alarm or a Sentry issue, always open or join an incident. Ace quiets an alert only when a **Known false positive** or **Suppressed pattern** has been confirmed several times and still has high confidence. The match is exact, meaning the same alert on the same workload or the same check on that workload, and a suppression never carries over to another workload or alert. For HTTP checks, a pattern quiets only a single, isolated failed check, because several failing together look like a real problem and Ace raises them as usual. If a quieted alert turns out to be real, Ace weakens the pattern and records a **Pattern that came back**. Any pattern Ace uses appears on the incident's **Memory** card. ## How memory fades Memories don't expire on a schedule. Instead, a learned pattern loses confidence when nothing confirms it, and each confirmation resets the clock. Suppressions fade fastest, so a stale one can't hide an alert for long. A suppression lapses within three weeks of its last confirmation and Ace raises the alert again, while other patterns fade more slowly. Contradicting evidence weakens a pattern right away, for example when an engineer closes an incident as **Expected behavior** after it was marked a real incident, or when a quieted alert turns into a real incident. Incident memories and fix outcomes don't fade, so Ace remembers a fix that made an incident worse for as long as the workload exists. ## Scope and isolation Every memory belongs to one organization, every read and write is limited to that organization, and nothing learned in one organization is used in another. A learned pattern applies to your whole organization, to one workload, or to one alert or check on a workload, and the **Memory** card shows **This organization**, **This workload** or **This workload and trigger**. Suppressions and fix history apply only to the workload they were learned on. Similar-incident search covers all your workloads, since the same failure often shows up on several services, and a result from another workload is labeled with that workload's name. When an admin deletes a workload, Ace deletes its incidents, their incident memories and the learned patterns scoped to it. ## Memory and operating rules [Operating rules](https://tryace.dev/docs/concepts/operating-rules) and memory both shape what Ace does: | | Operating rules | Memory | | --- | --- | --- | | Who writes it | Your admins, in **Settings** | Ace, from incidents, feedback and verified outcomes | | What it is | Instructions such as "Prefer rollbacks over forward fixes for the payments service" | Observations such as "this fix made things worse on this workload" | | Scope | Your organization, by kind of work (diagnosis, recommendations, pull requests, live operations) | Your organization, a workload, or one alert or check | | Changes over time | Only when an admin edits it | Strengthens, weakens and fades with evidence | | Can grant permissions | No | No | Both can only narrow what Ace does. Ace ignores an operating rule that asks for more, and memory never changes your action policy. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). ## Using memory from your coding agent The [Ace MCP server](https://tryace.dev/docs/mcp) gives coding agents and reviewers read-only access to the same memory. - `get_lessons` returns fixes that worked, fixes to avoid, and similar past incidents with the actions Ace took and how each was judged. - `search_incidents` finds incidents by meaning or by text. ```text Before you change anything for INS-1018-ab12, call get_lessons with that incident. Don't propose any fix listed under "Fixes to avoid". If a fix under "Fixes that worked" matches the root cause from get_incident_context, start from it. ``` ## Next steps - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): From first signal to closed incident, and how to read the results. - [Ace MCP server](https://tryace.dev/docs/mcp): Give your coding agent the same incident context and lessons. - [Operating rules](https://tryace.dev/docs/concepts/operating-rules): Tell Ace how your team wants the work done. - [Proactive detection](https://tryace.dev/docs/concepts/proactive-detection): The checks and watches that memory can quiet. --- Source: https://tryace.dev/docs/concepts/how-ace-works # How Ace works > What happens from the first signal to a closed incident, and how to read what Ace shows you on the incident page and in Slack. Source: https://tryace.dev/docs/concepts/how-ace-works Ace opens an incident when an alert fires on a workload and investigates it across the sources you connected. It proposes or takes a fix under your [action policy](https://tryace.dev/docs/concepts/actions-and-approvals), checks that the incident's own alerts recover, and then closes the incident. This page walks through each stage and how to read the results. ## What opens an incident A workload is a service or job you care about, such as an API, checkout or a background worker. Only alerts routed to a workload open incidents on it, and you choose them under **Which alerts start an investigation?** when you create the workload. | Source | What it is | | --- | --- | | Alerts from your tools | Monitors, alarms and issues from integrations such as Datadog, CloudWatch, Prometheus Alertmanager, Sentry or Grafana, routed to the workload. See [Integrations](https://tryace.dev/docs/integrations). | | Ace HTTP checks | Ace calls an endpoint on a schedule and opens an incident on an error rate spike, a latency regression or chronic slowness. See [Proactive detection](https://tryace.dev/docs/concepts/proactive-detection). | | Metric watches | Ace learns each watched series' normal range and opens an incident when a deviation persists. | | Early warnings | A forecast that a series will reach a known limit, or a capacity risk when it runs close to one. | | A person | **Run investigation now** on a workload, or `@Ace investigate ` in Slack. Both look at the last hour. | Change-only sources, such as deploys and feature flag or config changes, go on the workload's timeline. Ace weighs them in every investigation, but they never open an incident on their own. Duplicate alert deliveries are kept once. ### Severity An incident is critical, high, warning or info. For alerts from your tools, Ace maps the alert's severity label like this: | Alert label | Incident severity | | --- | --- | | `critical`, `page` | Critical | | `high`, `error` | High | | `warning`, `warn`, `medium`, `moderate` | Warning | | `info`, `none`, `low` | Info | | Missing or unrecognized | High | If an alert with a higher severity joins an incident, the incident's severity goes up. ## Grouping: one incident per workload While a workload has an open incident, new alerts on it join that incident, so an alert storm becomes one incident. Each alert is listed under **Signals** with an occurrence count, and the incident resolves when none of them is still firing. Alerts that join an incident don't each post to Slack. Instead the card updates when severity goes up, so a storm doesn't flood your channel. Manual investigations and forecasts always stay separate incidents. ### Linked incidents When one problem hits several workloads, such as a slow database behind several services, Ace opens one incident per workload and links a later incident to the first when they share an alert, or when the later one's cause sits in the first one's blast radius on the System Map. The later incident shows **Part of** the first, and the first shows **Also affecting**, on the incident page and in Slack. The incidents stay separate, each with its own alerts, diagnosis and close. While the first incident's fix is in progress, the linked incident holds back its own fix so the two don't race. ## The investigation ### When Ace investigates Ace investigates high and critical incidents on its own. Lower-severity incidents are recorded and watched, and Ace investigates them if they escalate or the alert is strong, such as an error rate spike on an HTTP check. It investigates again when something material changes, for example when severity goes up, the incident grows or a fix doesn't work. You can also start an investigation at any time: - **Run investigation now** or **Investigate** on a workload - **Investigate again** on an incident, with optional context such as "it's not the deploy, check the queue" - `@Ace investigate ` in Slack ### What Ace reads Each investigation starts from the incident's first alert and reads: - the sources chosen under **Which sources does Ace read?** for the workload, plus any source that routes its alerts, with a source that has nothing relevant recorded as such - deploys and flag, config and infrastructure changes before the incident, from every connected source that reports changes - the System Map, with the workload's services and their dependencies - the repositories linked to the workload - [memory](https://tryace.dev/docs/concepts/memory): similar past incidents, the workload's fix history and recently rejected recommendations - notes on the incident and the context you gave with **Investigate again** - your [operating rules](https://tryace.dev/docs/concepts/operating-rules), which shape the diagnosis and recommendations ### What an investigation produces | Part | What it is | | --- | --- | | Root cause | One hypothesis, or "inconclusive" when the evidence doesn't settle it. | | Confidence | High, medium or low. | | Category | **Code defect**, **Deploy regression**, **Config or flag change**, **Infrastructure change**, **Capacity saturation**, **Upstream dependency**, **Data or state**, or **Cause not established**. | | Supporting evidence | The findings, per source, that back the root cause. | | Contradicting evidence | Evidence that cuts against the causal path. | | Other causes considered | Alternatives, each with the reason it was ruled out. | | Open questions | What Ace couldn't establish. | | Contributing factors | What made the incident worse or more likely. | | How it connects | The path from change to effect to symptom, over System Map dependencies. | | Recommended actions | Steps such as a scaling change, config tuning, a rate-limit change, traffic shaping, capacity planning, a code change or an infrastructure-as-code change. Each comes with a rationale, the evidence behind it and the alert that will show it worked. | ## Reading the incident page The incident page has four sections, **Overview**, **Remediation**, **Evidence** and **Signals**, and the bar at the top shows the time since detection at each stage: **Detected**, **Investigating**, **Root cause**, **Remediation**, **Verifying**, then **Resolved** or **Closed**. ### The root cause card The card shows: - **Root cause**, or **Most likely cause** with **Inconclusive** when Ace couldn't settle it - a three-bar meter with **High confidence**, **Medium confidence** or **Low confidence** - whether the diagnosis is **Preliminary** or **Final**, plus **Updating**, **Based on earlier evidence** or **Recorded before recovery** if newer evidence has arrived. Check a preliminary diagnosis before you act on it, because it can still change. - how many sources Ace checked, the category, the headline and what happened - **How it connects**, with the change, effect and symptom each linked to its evidence - **Other causes Ace considered**, with the reason each was ruled out While an investigation runs, the card says "Correlating the signal with your workload, recent changes and connected sources." If the investigation fails, it says **The investigation didn't finish** and gives the reason. Coding agents can read the full result, including contradicting evidence and open questions, with `get_incident_context` on the [Ace MCP server](https://tryace.dev/docs/mcp). ### What Ace is doing, and when it needs you The **What Ace is doing** card shows Ace's goal, each attempt with its state (**In progress**, **Waiting**, **Done**, **Will retry** or **Asked**) and when Ace looks again. When Ace needs a decision, the card becomes **Ace needs you** and offers the answers that apply: **Review the step**, **I deployed it**, **I applied it**, **Investigate again** or **Close incident**. ### The side panel The side panel shows status, severity, workload, what detected the incident, who started it, start time and duration, occurrences, people and links. **Sources Ace read** marks each source **Findings**, **Nothing relevant** or **Skipped**. Notes you add appear under the incident, and Ace uses them in its next investigation after removing sensitive values. The **Memory** card shows what Ace brought from past incidents and what it learned from this one. ## Slack updates When Slack is connected, Ace posts each incident as a card and updates it in place: - While the incident is open, the title shows severity and workload, and once it's resolved the title says the workload recovered. - The buttons are **I'm on it** to acknowledge, **Evidence** and **View suggested fix**. **Evidence** opens the root-cause type, other causes considered, grouped alerts and the timeline. - The footer has **View workload** and the incident reference, such as `INS-1018-ab12`. - Linked incidents show **Part of** or **Also affecting**. The RCA goes in the thread, titled **RCA** or **Final RCA**, or marked preliminary, with the confidence and hypothesis, key evidence, what happened, contributing factors and the recommended next step. Ace asks for decisions with buttons in the incident's thread. If nobody answers, Ace reminds the thread, and on high and critical incidents it also mentions people who can answer. Questions about early warnings stay in Ace and aren't posted. Ace asks only in Slack threads and on the incident page, and it doesn't page anyone through PagerDuty or incident.io, which are read-only integrations. ### Talking to Ace in Slack - `@Ace investigate ` shows a confirmation only you can see, with **Start investigation** and **Cancel**. - A reply in an incident's thread, such as "it's not the deploy, check the queue", makes Ace investigate again with your words as context. - A request for an action, such as `@Ace scale checkout to 6 tasks`, makes Ace build a plan from live state and post it for approval. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). Slack actions need your Slack user linked to your Ace account, so Ace can check your permissions. ## Fixing the incident Ace proposes a fix that matches the cause: - undoing a correlated change, such as a deploy, a flag or config change, or an infrastructure change - an operation on a running system for causes outside code, such as scaling a service or failing over - a pull request for a code defect Anything your [action policy](https://tryace.dev/docs/concepts/actions-and-approvals) doesn't allow automatically becomes a question to a person. After recovery, Ace can propose redoing work the incident lost, such as redriving an SQS dead-letter queue or replaying Trigger.dev runs. These steps wait for the fix to deploy, so the replayed work doesn't hit the same failure, and they always need approval. They never hold the incident open. ## Recovery verification Ace closes an incident only on evidence that the workload recovered: 1. The change is live. A merged fix counts once it's deployed, which Ace confirms from your deploy provider or asks you to confirm with **I deployed it**. 2. The incident's own alerts recover. The original alert or monitor resolves, fresh HTTP checks are healthy, or a measured series returns to its normal level and stays there. 3. Ace keeps watching for the same failure after recovery. Passing CI, a merged pull request or someone closing the incident never counts as recovery, because none of them show the workload is healthy. If the workload recovered before the change was applied, or after some other change, Ace records that it **recovered on its own**, and that result doesn't count toward earned autonomy. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). Use **I applied it** to tell Ace about a change you made yourself, with the time you made it. That time can't be in the future or more than a day before the incident began. Ace verifies recovery on the incident's alerts and never runs or undoes your change. ## Closing and reopening Ace closes an incident on its own when one of these happens: - the alerts that opened it resolve - its HTTP checks recover - Ace verifies that an action recovered the workload - for alerts with no "recovered" event, such as errors or failed jobs, the alert stays quiet after a deploy An early warning closes when its condition clears. A metric-watch incident also closes when the watch stops, for example when someone mutes or deletes it, it runs out of data, or Ace learns the new level, but that close doesn't count as a recovery. ### Closing an incident yourself Select **Close** on the incident page and choose a reason under **Why close?**: | Reason | Use it when | | --- | --- | | **Incident resolved** | It was a real incident and it's over. | | **Expected behavior** | The alert reflects normal behavior. | | **Handled elsewhere** | Your team deals with it somewhere else. | | **Not actionable** | There's nothing to do about it. | | **Duplicate investigation** | Another incident already covers it. | Closing ends any running investigation, and Ace remembers your reason for that alert on that workload. See [How memory works](https://tryace.dev/docs/concepts/memory). ### Reopening If Ace closed the incident and the same alert fires within 24 hours, the incident reopens, and after 24 hours a new incident opens instead. If a person closed the incident, a new incident always opens. ### What happens on close On any close, Ace marks the firing alerts resolved, finalizes the diagnosis and withdraws proposed steps that no longer apply. It records what it learned in [memory](https://tryace.dev/docs/concepts/memory) and refreshes the Slack card and the cards of linked incidents. ## Next steps - [Proactive detection](https://tryace.dev/docs/concepts/proactive-detection): The checks, watches and early warnings that open incidents before your alerts do. - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): What Ace can change, who approves it and how recovery is verified. - [How memory works](https://tryace.dev/docs/concepts/memory): What Ace learns from each incident and how it recalls it. - [Ace MCP server](https://tryace.dev/docs/mcp): Give coding agents the full incident context. --- Source: https://tryace.dev/docs/concepts/proactive-detection # Proactive detection > How Ace watches your workloads on its own with HTTP checks, metric watches and early warnings, how it learns what normal looks like, and how to tune it. Source: https://tryace.dev/docs/concepts/proactive-detection Besides your own alerts, Ace raises three kinds of alerts itself. Ace HTTP checks call your endpoints, metric watches read your providers' metrics, and early warnings fire when a series is heading toward a known limit. Watches cover latency, errors, throughput, saturation and cost. Ace learns each series' normal range and opens an incident when a deviation persists. Most of this needs no setup, because Ace builds the watches from your System Map and connected sources. ## Ace HTTP checks An Ace HTTP check calls one of your endpoints on a schedule and measures latency and errors. Add one when you create a workload, with **HTTP health check** under **Which alerts start an investigation?**, or later in the workload's settings under **Investigation triggers**. Members and admins can both add HTTP checks. | Setting | What you set | | --- | --- | | **Endpoint** | **Base URL**, **Path**, **Method** and **Authentication** (**None**, **API Key Header** or **Bearer Token**). Credentials are stored securely. | | **Request payload** | The request body, for methods that send one. | | **Frequency & limits** | How often Ace checks, from the frequencies available to your workspace: **Light** (4 per hour), **Standard** (12 per hour), **Intensive** (30 per hour) or **Aggressive** (60 per hour). To stay within your API's limits, turn on **Rate limits** and set **Max checks per hour** and, if you want, **Max checks per day**. | | **Anomaly detection** | **Adaptive baseline** adjusts thresholds to the traffic Ace observes. Turn it off to set a **P95 latency threshold (ms)** and an **Error rate threshold (%)** yourself. | Ace confirms a failed check with follow-up checks before it raises anything, so a single blip doesn't open an incident. A failure that holds opens an **Error Rate Spike**, **Latency Regression** or **Chronic Slowness** incident on the workload. Ace investigates high and critical HTTP check incidents on its own, and for error rate spikes warning severity is enough. Failures that clear on their own teach Ace that the check is noisy. See [How memory works](https://tryace.dev/docs/concepts/memory). ## Metric watches A metric watch is one series that Ace reads from a provider on a schedule and compares with its learned normal range. | What it measures | Read | | --- | --- | | Latency, error rate, throughput, failures | Every 5 minutes | | Saturation and queue age | Every 15 minutes | | Cost | Daily | ### Which series Ace watches Ace creates watches for three sets of resources: - the resources a workload names exactly on the System Map, from the resources, names, tags and repositories you chose when you created it - what those resources contain - resources one runtime dependency away, such as the workload's database, cache or queue Ace never matches resources by a loose name match, which would put watches on unrelated resources, and it creates watches only on connections the workload's investigations read. When the System Map changes, Ace updates the list and keeps your tuning and mutes, and a watch whose resource leaves the workload stops running. The tables below list every watch Ace can create, where "Limit" is the known ceiling that early warnings measure against. #### AWS (CloudWatch) | Resource | Watches | Limit | | --- | --- | --- | | RDS instance | CPU, Database connections, Freeable memory (a drop is bad), Free storage (a drop is bad), Read latency, Write latency | CPU 100%, connections at the instance's maximum (when known), free storage 0 | | Aurora cluster | CPU, Database connections, Freeable memory (a drop is bad), Serverless capacity (ACU, when the maximum capacity is known) | CPU 100%, connections at the maximum, ACU at the maximum capacity | | RDS Proxy | Client connections, Database connections, Connection borrow latency | None | | ElastiCache cluster | Cache hit rate (a drop is bad), Engine CPU, Memory used, Connections, Evictions | CPU and memory 100%, connections at `maxclients`, or 65,000 by default | | Lambda function | Duration p95, Errors, Invocations (a drop is bad), Throttles, Concurrent executions (when reserved concurrency is set) | Concurrency at the reserved concurrency | | ECS service | CPU, Memory | 100% | | SQS queue | Age of oldest message, Messages visible | Message age at the queue's retention period (4 days by default) | | Application Load Balancer | Target response time p95, Target 5xx responses, Requests (a drop is bad) | None | | DynamoDB table | Read throttle events, Write throttle events, System errors | None | | EC2 instance (not in an Auto Scaling group) | CPU, Status check failed | CPU 100% | | Auto Scaling group | In-service instances, when the group's maximum size is known | The group's maximum size | | Kinesis stream | Iterator age, Write throughput exceeded | Iterator age at the stream's retention period (24 hours by default) | | API Gateway REST API | Latency p95, 5xx errors | None | | CloudFront distribution | 5xx error rate | None | | Each AWS service above | The account's daily spend on that service | None | Ace watches an Auto Scaling group as a whole, without separate watches on its instances. Daily spend comes from AWS billing metrics, which exist only when billing alerts are turned on in the account. > **CloudWatch watches are off until an admin turns them on** > > CloudWatch metric reads are billed to your AWS account, so CloudWatch watches are listed but not read until an admin turns on **Watch CloudWatch metrics**. The switch is under **CloudWatch watches** in **Settings** > **General** and in the AWS settings, and a notice in a workload's **Watched signals** links to it too. > > Watches on other providers run either way. See [AWS](https://tryace.dev/docs/integrations/aws). #### Kubernetes (through Prometheus) Ace watches Deployments, StatefulSets and DaemonSets through your organization's Prometheus source, which is a Grafana Prometheus data source or Amazon Managed Prometheus. The organization must have exactly one Prometheus source, so Ace knows which one to query. | Watch | Read | Limit | | --- | --- | --- | | CPU of limit | Every 15 minutes | 100% | | Memory of limit | Every 15 minutes | 100% | | Container restarts | Every 5 minutes | None | | Unavailable replicas | Every 5 minutes | None | #### Other integrations | Integration and resource | Watches | Limit | | --- | --- | --- | | Supabase project | Postgres connections, CPU load, Memory used, Disk used | Connections at the project's maximum, and 100% for CPU, memory and disk | | New Relic service | Latency p95, Error rate, Throughput (a drop is bad) | None | | New Relic host | CPU, Memory used, Disk used | 100% | | Honeycomb dataset | Latency p95, Error rate | None | | Cloudflare zone or proxied DNS record | Edge 5xx share | None | | Trigger.dev task | Failed runs | None | | Inngest function | Failed runs | None | | LiteLLM gateway | Spend, Failure rate, Latency p95 | None | | LiteLLM model | Failure rate, Latency p95 | None | | PostHog project | Pageviews (a drop is bad), Exceptions | None | | Datadog APM service | Latency and errors for each of the service's trace metrics on the System Map | None | ### Your own watches You can add a watch for any other series, such as a business metric: 1. In the workload's **Watched signals**, choose the source and connection. 2. Write the query, name the watch and pick what it measures. 3. If a drop is the problem, turn on **A drop is bad (hit rate, traffic, free storage)**. 4. Select **Preview** to read the last hour and check the query. 5. Select **Add watch**. | Source | What you enter | | --- | --- | | Amazon CloudWatch | **Namespace**, **Metric**, dimensions, **Statistic** and **Region** | | Datadog | **Metric query** | | New Relic | **Account ID** and an **NRQL aggregate** | | Honeycomb | **Dataset**, **Calculation** and column | | Grafana | **Prometheus datasource UID** and **PromQL** | | Amazon Managed Prometheus | **PromQL** | | Supabase | **Project ref** and **Metric** (connections, CPU, memory or disk) | | Cloudflare | **Host** | | Trigger.dev | **Task** | | Inngest | **Function** | | LiteLLM | **Metric** (spend, requests, failure rate or latency p95) and, optionally, **Model** | | PostHog | **Event** | A workload can have up to 50 of your own watches, and only admins can add or delete them. ## How Ace decides a series is abnormal Ace learns each watch's normal range from the series' own history, including daily and weekly patterns, so a nightly batch job or a Monday-morning peak isn't flagged. A new watch shows **Learning** until Ace has enough history to judge it. Where the provider keeps history, Ace reads it first, so many watches can alert right away, though old deviations in that history never open anything. **Start alerting now** ends learning early. One odd sample doesn't fire a watch, because the deviation has to hold. The watch clears once the series is back to normal, and Ace ignores changes too small to matter, such as a tiny move on a very steady series. A step change fires. If the series stays at the new level, Ace eventually learns it as normal and the incident closes without counting as a recovery. Slow drift moves the normal range with it, so a forecast against a known limit is what catches it. A sample past a threshold you set always counts as deviating, even while the watch is learning. | Severity | When | | --- | --- | | High | Past your threshold, or far from normal. Ace investigates on its own. | | Warning | Any other firing deviation. Recorded on the incident and watched. | Like any alert, a firing watch opens an incident on the workload or joins the open one. See [How Ace works](https://tryace.dev/docs/concepts/how-ace-works). ## Early warnings Any watch with a known limit in the tables above can raise an early warning before anything fails. | | Forecast | Capacity risk | | --- | --- | --- | | Raised when | The series' trend is on course to reach its limit soon | The series is running close to its limit | | Severity | Higher the sooner the limit will be reached | Higher the closer the series is to its limit | | Clears | When the trend no longer projects a breach | When the series moves back away from its limit | | In Ace | **Forecast** with "Reaches its limit" and the time | **Capacity risk** with "At N% of its limit" | When a forecast comes true, it becomes a capacity risk and keeps its investigation. At warning or above, an early warning starts an investigation and is posted once to your alert channels, while **Info** forecasts stay in Ace. If the workload already has an open incident, Ace adds the early warning to it as evidence. Ace never acts on its own on an early warning, because nothing has failed yet. Any preventive step waits for a person, whatever your [action policy](https://tryace.dev/docs/concepts/actions-and-approvals) says. A workload's open early warnings are listed in its **Watched signals**, and the [Ace MCP server](https://tryace.dev/docs/mcp) returns them from `get_workload_context`. ## Tuning detection A workload's **Watched signals** settings list each watch with what it measures, how often it's read, and its current value against its normal range and its limit or threshold. Each watch has one of these statuses: | Status | Meaning | | --- | --- | | **Normal** | Within its learned normal. | | **Learning** | Building its baseline. | | **Waiting for data** | No sample read yet. | | **Deviating** | Firing now. | | **Can't read** | The last reads failed. The error is shown. | | **Muted** | Turned off. | **Checked less often** means your workspace hit its read limit for that provider, so Ace reads the watch less often until that clears. Open a watch to change it: | Control | What it does | | --- | --- | | **Sensitivity** | **Low**, **Normal** (the default) or **High**. Low needs a bigger deviation to fire, and High catches smaller ones. | | **Alert above** / **Alert below** | A fixed threshold, with **Save threshold**. A sample past it always counts as deviating, even while the watch is still learning. | | **Start alerting now** | Ends learning early for this watch. **Start alerting for all** does it for every learning watch on the workload. | | Mute | Stops the watch. If the watch is part of an open incident, muting ends its alert, and the incident closes if nothing else in it is still firing. | | **Delete** | For your own watches, admins only. "Ace stops watching this series and deletes its history. Incidents it opened are kept." | Members and admins can tune and mute watches. Only admins can add or delete custom watches. To stop every watch, an admin can turn on **Pause all metric watches** in **Settings** > **General**. Ace then stops reading your providers for every workload's watches and closes what they raised, and the watches restart when you resume. Closing a watch's incident as **Expected behavior** or **Not actionable** also tunes it, and after enough of these closes Ace stops raising that alert on that workload. See [How memory works](https://tryace.dev/docs/concepts/memory). ## Next steps - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): What happens after a check, watch or early warning opens an incident. - [How memory works](https://tryace.dev/docs/concepts/memory): How repeated closes quiet a noisy signal, and how that fades. - [Integrations](https://tryace.dev/docs/integrations): Connect the sources Ace builds watches from. --- Source: https://tryace.dev/docs/concepts/actions-and-approvals # Actions and approvals > Every change Ace can make, how your action policy decides what needs a person, who can approve, how undo works, how Ace verifies recovery, and how earned autonomy lets an action run automatically. Source: https://tryace.dev/docs/concepts/actions-and-approvals Ace can roll back a bad deploy, scale a service, flip a feature flag or open a pull request. Each change is an action with an exact plan, and your action policy decides whether a person approves the plan or Ace runs it on its own. Every action needs approval by default, and after any change Ace checks that the workload actually recovered. ## How an action works 1. Ace prepares each plan from live state, naming the exact change, the resource it touches, a preflight check, how to undo it (when it can) and the alert that will show it worked. 2. Your policy decides whether the plan waits for approval or runs automatically. 3. A person approves the exact plan on the incident page under **Remediation**, or in Slack. If you choose **Edit values** and change a value, such as the replica count, Ace prepares the plan again and it needs a new approval, because an approval covers only the exact plan. 4. Ace applies the change and confirms it took effect. 5. Ace verifies recovery on the incident's own alerts, then keeps watching in case the problem comes back. ### Where actions come from - **Undoing a correlated change.** When a deploy, a flag or config change, or an infrastructure change lines up with the incident, Ace proposes undoing it. If a person made that change during or shortly after an incident on the workload, Ace always asks before undoing it, whatever the policy. - **Operations on running systems**, such as scaling a service or failing over a load balancer, for causes outside code. Ace doesn't propose these on a low-confidence diagnosis, since a live change aimed at the wrong cause can make things worse. - **Fix pull requests** for code defects, for your team to review. - **Work after the fix.** Once the workload recovers, Ace can propose redoing lost work, such as redriving an SQS dead-letter queue or replaying failed runs, and these always wait for the fix to deploy. - **Requests in Slack**, such as `@Ace scale checkout to 6 tasks`. Ace picks the matching operation for the workload, prepares it from live state and posts it for approval, and `@Ace help ` lists what you can ask for there. ### Three kinds of actions | Kind | What it does | How it runs | | --- | --- | --- | | Remediation | Changes the workload to end the incident. | Under your action policy. Judged on recovery. | | Observability update | Changes only what people see about the incident, such as annotations, silences, downtimes, mutes and issue state. | Automatically by default once allowed on the connection, unless a policy for that exact action says otherwise. Must be undoable. Never judged on recovery. | | After the fix | Redoes work the incident lost. | Waits for the fix to deploy and always needs approval. Never holds the incident open. | ## Before Ace can act Ace's access is read-only until an admin allows changes, and each integration has its own switch: | Integration | How an admin allows actions | | --- | --- | | GitHub | The Ace GitHub App opens revert and fix pull requests. **Roll back deploy** and **Redeploy** run through a deploy workflow that an admin confirms for each workload under **Rollback workflow** in **Settings** > **Actions** > **Workloads**. | | Kubernetes | The Ace connector in your cluster makes changes, only in the namespaces confirmed for the workload. See [Kubernetes](https://tryace.dev/docs/integrations/kubernetes). | | AWS | An optional actions role with the operation groups you choose. Each group is off until you choose it, and actions run only on the resources an admin confirms for each workload. See [AWS](https://tryace.dev/docs/integrations/aws). | | Other connected tools | Turn on **Allow Ace to run approved actions** on the connection. For tools whose only writes are observability updates, the switch is **Allow Ace to update this tool during incidents**. Ace never proposes an action that the connection's credentials are known to lack permission for. | PagerDuty and incident.io have no actions. ## Action policy An admin sets the policy in **Settings** > **Actions** > **Workloads** by choosing a workload, an action and a mode. | Mode | In Ace | What happens | | --- | --- | --- | | Off | **Off**, "Ace never runs this action." | Ace never runs it, but can still recommend the fix. | | Approval | **Approval**, "A person approves each plan." | Every plan waits for approval. | | Approval, tracking eligibility | **Approval** with **Track eligibility** on, "Record whether each proposal would have qualified for Automatic." | Every plan waits for approval, and Ace records whether each one would have qualified for Automatic. This is the default when no policy is set. | | Automatic | **Automatic**, "Ace acts and posts an Undo option in Slack." | Ace runs qualifying plans on its own after a countdown in Slack. Locked until the action qualifies. See [Earned autonomy](#earned-autonomy). | ### Which policy applies - **Stop all actions** in **Settings** > **Actions** sets the whole organization to Off, while Ace still investigates and recommends fixes. **Resume actions** turns actions back on. - If any policy that matches the action is Off, the action doesn't run. - Otherwise the most specific policy wins. A policy for the exact action beats one for all actions, then a policy for the workload beats one for all workloads, and a policy for the environment beats one for all environments. - **Change freezes**, in **Settings** > **Actions** > **Change freezes**, are periods such as a launch when nothing runs automatically, though Ace still recommends fixes for approval. You can schedule up to 20. ## Approvals ### Who can approve | Plan | Who can approve it | | --- | --- | | A change to production | An admin, or a member an admin granted approval for that action (or all actions) in that environment (or all environments). | | Any other plan | Any member. | In any environment, Ace treats these as production changes: - every change it makes through Kubernetes, AWS or your other connected tools, except observability updates - every rollback and redeploy - every undo of a correlated change Fix pull requests and changes you report with **I applied it** count as production when the environment is production. Admins can approve every action in every environment. To choose what a member can approve, use **Settings** > **Team** or set it when you invite them. ### Two approvers Each plan needs one approver by default. To require **Two people**, add an approval rule in **Settings** > **Actions** > **Approvals**. A rule can cover all actions or one, all workloads or one, and all environments or one, and the most specific rule applies. The two approvals must come from different people, and the second approver sees **Approve as second**. Actions with two approvers never run automatically, because the rule asks for two people. ### Approving on the web and in Slack Approve on the incident page under **Remediation** or from the decision card Ace posts in the incident's Slack thread, since both approve the same plan. The Slack decision card has these buttons: - **Approve**, labeled with the action, such as **Approve rollback** or **Create revert PR** - **Edit values** - **Decline** - Ace's alternatives, as **Instead: …** - **Something else…**, to tell Ace what you want in your own words Acting from Slack needs your Slack user linked to your Ace account, because Ace checks your permission when you click. If nobody answers, Ace reminds the thread, and on high and critical incidents it also mentions people who can approve, admins first. When a fix spans several repositories, its pull requests are approved together in one decision. ## Earned autonomy Ace earns the right to act on its own one action, workload and environment at a time. **Automatic** stays locked for an action on a workload until the action has a verified track record there, and your policy, preflight checks and daily limits still apply after it unlocks. The workload's action settings show its **Track record**, such as "3 of 5 verified recoveries, 0 worsened", and why Automatic is locked. ### To turn on Automatic - The policy names one action, one workload and one environment, and the workload's deployment mapping for that environment is confirmed. Automatic can't be set for **All actions** or **All workloads**, because a track record only means something for one action on one workload. - The action has at least the number of verified recoveries set in **Recoveries required** (5 by default, 1 to 100), with no outcome that made things worse, all in that exact workload, environment and deployment mapping. - Only recoveries after a deploy that the provider recorded count. A deploy you confirmed with **I deployed it**, or a change you reported with **I applied it**, never counts, since Ace can't verify those itself. - For **Kubernetes change** (an exact patch), a repair grant replaces the track record. An admin approves the exact fields Ace may change in the confirmed namespaces, with an expiry, under **Settings** > **Data sources** > **Kubernetes**. Only reversible changes to approved values qualify, and creates, deletes, cluster-wide changes, credential fields and list edits always need a person. If a later outcome makes things worse, the track record stops qualifying and the action goes back to needing approval. ### When an automatic action runs A plan runs on its own only if Automatic is on and all of these hold. If any one fails, Ace posts the plan for approval with the reason: - The incident isn't an early warning. - The action doesn't write to a data store. - No approval rule asks for two approvers. - It's inside the policy's set hours (**Limit to set hours**) and outside any change freeze. - The incident isn't holding its fix for a [linked incident](https://tryace.dev/docs/concepts/how-ace-works). - The preflight is safe. A blocked preflight never runs, and one with cautions runs only if **Run despite preflight cautions** is on. - The action doesn't always need a person. See the [catalog](#action-catalog). - For Kubernetes, no GitOps controller (Argo CD or Flux) manages the object, since the controller would revert the change. The operation also isn't one that always needs a person, such as an image from a registry the workload doesn't already use. - For everything else, Ace tied the incident to a specific deployment, unless **Run without a linked deployment** is on. - The plan can be undone. - The action is under its **Daily limit** (5 by default, up to 100, counted per UTC day). AWS plans always carry a preflight caution, so AWS actions run automatically only with **Run despite preflight cautions** on. Before an automatic action runs, Ace posts a short countdown with an **Undo** option in the incident's Slack thread. If the countdown can't be posted, the action doesn't run, because nobody would have the chance to stop it. For **Fix pull request** and **Revert pull request**, you can turn on **Auto-merge pull requests**, which is off by default. With it on, Ace merges when every Ace reviewer passes and checks are green, after announcing the merge in Slack so anyone can stop it. ## Undo and rollback Undo exists only where Ace can restore the exact prior state, and the catalog shows which actions have it. - Before an automatic action runs, **Undo** in the Slack countdown cancels it. - After an action is applied, undo is a new action that Ace prepares from the prior state, and it needs approval like any other. - Observability updates are undone with one click on the incident. The click is your approval. When a change makes things worse, Ace can propose undoing it but never undoes it automatically. **Roll back deploy** redeploys a known-healthy revision through the deploy workflow an admin confirmed for the workload, without reversing database changes or provisioning infrastructure. A rollback can include the matching revert pull request in the same approval, so your main branch matches what's running. ## Recovery verification Ace first confirms the action took effect, then checks the alerts that opened the incident, such as the original HTTP check, the same alert from your tool, or the metric's normal range. Passing CI, a merged pull request or someone closing the incident never counts, because none of them show the workload is healthy. See [How Ace works](https://tryace.dev/docs/concepts/how-ace-works). | Result | In Ace | What it means | Counts toward Automatic | | --- | --- | --- | --- | | Verified recovered | **Recovered** | The incident's alerts recovered after the change took effect. | Yes, when the provider recorded the deploy | | Did not recover | **Did not recover** | The change took effect, but the alerts didn't recover. | No | | Made it worse | **Made it worse** | The alerts got worse after the change. | Blocks Automatic for that action on that scope | | Recovered on its own | **Recovered on its own** | The workload recovered before the change took effect, or after some other change. | No | After recovery, Ace keeps watching in case the problem returns. Results feed [memory](https://tryace.dev/docs/concepts/memory), so a fix that made an incident worse on a workload is never proposed again there, and a fix that didn't recover it is proposed again only with new evidence. ## Action catalog Each operation below is a separate action with its own policy and track record, named as Ace shows it. - **Always a person** means it never runs automatically, whatever the policy. "Yes (data store)" marks a write to a data store. - **Undo** means Ace can restore the exact prior state. - Values you can edit before approving, such as a replica count, show their allowed range. ### GitHub and deploys | Action | What it does | Always a person | Undo | | --- | --- | --- | --- | | Fix pull request | Opens a pull request with a fix for a code, config or infrastructure-as-code cause. | No | Yes | | Revert pull request | Opens a pull request that reverts the change behind the incident. | No | Yes | | Roll back deploy | Redeploys a known-healthy revision through your confirmed deploy workflow. | No | Yes | | Redeploy | Deploys the revision already serving again, with no code change. | No | No | | Applied by a person | Records a change you made yourself (**I applied it**). Ace runs nothing and only verifies recovery. | Yes | No | ### Kubernetes | Action | What it does | Always a person | Undo | | --- | --- | --- | --- | | Kubernetes change | An exact patch, create or delete on a namespaced resource. Automatic only with a repair grant. | No | No | | Kubernetes rollout restart | Restarts a Deployment, StatefulSet or DaemonSet's pods. | No | No | | Kubernetes scale | Sets replicas on a Deployment or StatefulSet (0–1,000). | No | Yes | | Kubernetes autoscaler update | Changes an autoscaler's minimum and maximum replicas (1–1,000) or CPU target (1–1,000%). | No | Yes | | Kubernetes rollout undo | Restores a previous revision. | No | Yes | | Kubernetes rollout pause | Pauses a Deployment's rollout. | No | Yes | | Kubernetes rollout resume | Resumes a paused Deployment rollout. | No | Yes | | Kubernetes container image | Sets a container's image, pinned by digest. | No | Yes | | Kubernetes container resources | Changes a container's CPU and memory requests and limits. | No | Yes | | Kubernetes environment value | Sets a non-secret environment variable on a container. | Yes | No | | Kubernetes CronJob suspend | Suspends a CronJob. | No | Yes | | Kubernetes CronJob resume | Resumes a suspended CronJob. | Yes | Yes | | Kubernetes Job re-run | Runs a Job, or a CronJob's job, again. | Yes | No | | Kubernetes disruption budget | Changes a PodDisruptionBudget. | No | Yes | | Kubernetes node cordon | Stops new pods from scheduling on a node. | Yes | Yes | | Kubernetes node uncordon | Lets pods schedule on a node again. | Yes | Yes | | Kubernetes node taint | Adds a taint to a node. | Yes | Yes | | Kubernetes node taint removal | Removes a taint from a node. | Yes | Yes | | Kubernetes pod delete | Deletes a pod so its controller replaces it. | Yes | No | | Kubernetes node drain | Cordons a node and evicts the confirmed namespaces' pods, respecting disruption budgets. Undo only uncordons. | Yes | No | | Argo Rollouts abort | Aborts an Argo Rollout. | No | Yes | | Argo Rollouts promote | Promotes an Argo Rollout. | Yes | No | | Argo Rollouts retry | Retries an aborted Argo Rollout. | Yes | Yes | | Argo Rollouts undo | Restores an Argo Rollout's previous revision. | No | Yes | | Argo CD sync | Syncs an Argo CD application to what Git holds now. | Yes | No | | Argo CD rollback | Rolls an Argo CD application back to an earlier history entry. Needs auto-sync off. | No | Yes | | Argo CD auto-sync disable | Turns off auto-sync for an application. | Yes | Yes | | Argo CD auto-sync enable | Turns on auto-sync for an application. | Yes | Yes | | Flux suspend | Suspends a Flux Kustomization or HelmRelease. | Yes | Yes | | Flux resume | Resumes a Flux Kustomization or HelmRelease. | Yes | Yes | | Flux reconcile | Asks Flux to reconcile a Kustomization or HelmRelease now. | No | No | Any Kubernetes change to an object a GitOps controller manages always needs a person. ### AWS Each AWS action belongs to an operation group that you turn on in the actions role, and all of them run only on resources confirmed for the workload. | Action | What it does | Always a person | Undo | | --- | --- | --- | --- | | ECS service rollback | Returns an ECS service to its previous task definition. | No | Yes | | ECS service restart | Forces a new deployment of an ECS service. | No | No | | ECS desired task count | Sets a service's desired tasks (0–5,000). | No | Yes | | ECS task stop | Stops one ECS task so the service replaces it. | No | No | | ECS auto scaling limits | Changes a service's scaling minimum and maximum (0–5,000). | No | Yes | | ECS task size | Changes a task's CPU (128–16,384 units) and memory (512–122,880 MiB). | Yes | Yes | | Lambda provisioned concurrency scaling limits | Changes provisioned-concurrency scaling limits (0–10,000). | No | Yes | | Lambda alias rollback | Points an alias back to its previous version. | No | Yes | | Lambda alias weighted routing | Shifts traffic between alias versions (weight 0–1). | No | Yes | | Lambda reserved concurrency | Sets reserved concurrency (0–10,000). | No | Yes | | Lambda provisioned concurrency | Sets provisioned concurrency (1–10,000). | No | Yes | | Lambda event source mapping enable/disable | Turns an event source mapping on or off. | No | Yes | | Lambda event source batch size | Changes an event source's batch size (1–10,000). | No | Yes | | Lambda memory and timeout | Changes a function's memory (128–10,240 MB) and timeout (1–900 s). | No | Yes | | Auto Scaling group desired capacity | Sets desired instances (0–10,000). | No | Yes | | Auto Scaling group size limits | Changes a group's minimum and maximum size (0–10,000). | No | Yes | | Auto Scaling process suspension | Suspends Auto Scaling processes. | No | Yes | | Auto Scaling process resumption | Resumes suspended processes. | No | Yes | | Auto Scaling instance refresh | Starts an instance refresh (minimum healthy 0–100%). | Yes | No | | Auto Scaling instance refresh cancellation | Cancels a running instance refresh. | No | No | | Auto Scaling instance refresh rollback | Rolls back a running instance refresh. | No | No | | Auto Scaling instance health | Marks an instance unhealthy so the group replaces it. | No | No | | Auto Scaling instance termination | Terminates one instance in a group. | No | No | | Auto Scaling group instance type | Changes a group's instance type. | Yes | Yes | | EC2 instance reboot | Reboots an instance. | No | No | | EC2 instance stop | Stops an instance. | Yes | Yes | | EC2 instance start | Starts a stopped instance. | Yes | Yes | | EC2 instance type | Changes an instance's type. | Yes | Yes | | EKS managed node group scaling | Changes a node group's minimum, maximum and desired size (0–1,000). | No | Yes | | RDS instance reboot | Reboots a database instance. | Yes (data store) | No | | Aurora cluster failover | Fails an Aurora cluster over to a replica. | Yes (data store) | No | | RDS instance class | Changes a database instance's class. | Yes | Yes | | RDS parameter | Changes a database parameter. | Yes (data store) | Yes | | Aurora reader added | Adds a reader to an Aurora cluster. | Yes (data store) | Yes | | Aurora reader removed | Removes a reader from an Aurora cluster. | Yes | Yes | | Aurora Serverless v2 capacity | Changes minimum and maximum capacity (0–256 ACU). | Yes (data store) | Yes | | RDS storage and autoscaling limit | Grows storage and changes its autoscaling limit (20–65,536 GiB). | Yes (data store) | Yes | | RDS Proxy connection pool | Changes the proxy's connection limits (1–100%). | Yes (data store) | Yes | | DynamoDB provisioned capacity | Changes read and write capacity (1–40,000 units). | Yes (data store) | Yes | | DynamoDB auto scaling limits | Changes a table's scaling limits (0–40,000 capacity units). | Yes (data store) | Yes | | DynamoDB capacity mode | Switches between provisioned and on-demand capacity. | Yes | Yes | | ElastiCache node reboot | Reboots cache nodes. | Yes (data store) | No | | ElastiCache replica count | Changes replicas per shard (0–5). | Yes (data store) | Yes | | ElastiCache node type | Changes the node type. | Yes | Yes | | ElastiCache failover | Fails a replication group over. | Yes | No | | SQS dead-letter queue redrive | After the fix: moves messages from the dead-letter queue back to the source queue (1–500 messages per second). | Yes (data store) | No | | SQS visibility timeout | Changes the visibility timeout (0–43,200 s). | No | Yes | | SQS maximum receives | Changes receives before dead-lettering (1–1,000). | No | Yes | | Kinesis shard count | Changes a stream's shards (1–10,000). | No | Yes | | Target group health check | Changes a target group's health check. | No | Yes | | Target group deregistration delay | Changes the deregistration delay (0–3,600 s). | No | Yes | | Load balancer target removal | Removes a target from a target group. | No | Yes | | Load balancer target registration | Adds a target to a target group. | No | Yes | | Load balancer traffic shift | Changes a listener's weights between target groups. | No | Yes | | Route 53 weighted record | Changes a weighted record's weight (0–255). | No | Yes | | Route 53 failover swap | Swaps primary and secondary failover records. | Yes | Yes | | CloudFront invalidation | Invalidates cached paths. | No | No | | API Gateway stage throttling | Changes a stage's rate (0–100,000 per second) and burst (0–100,000). | No | Yes | | API Gateway canary promotion | Promotes a stage's canary. Only a person proposes it. | No | No | | API Gateway canary rollback | Rolls back a stage's canary. | No | No | | WAF IP set addition | Adds addresses to a WAF IP set. | No | Yes | | WAF IP set removal | Removes addresses from a WAF IP set. | No | Yes | | WAF rule action override | Overrides a WAF rule's action. | No | Yes | | CodeDeploy stop and roll back | Stops a deployment and rolls it back. | No | No | | CloudFormation update cancellation | Cancels a stack update in progress. | No | No | | CloudFormation rollback continuation | Continues a stack's stuck rollback. | No | No | | AppConfig deployment stop | Stops an AppConfig deployment. | No | No | | Step Functions execution redrive | After the fix: redrives a failed execution. | No | No | | EventBridge rule disable | Disables an EventBridge rule. | No | Yes | | EventBridge rule enable | Enables an EventBridge rule. | No | Yes | | SSM Automation runbook | Runs an SSM Automation document an admin allowed. | Yes | No | | SSM Run Command | Runs an SSM command on instances. | Yes | No | | CloudWatch alarm actions muted | Observability update: mutes a firing alarm's actions during an approved fix. Runs on its own. | No | Yes | | CloudWatch alarm actions restored | Observability update: restores an alarm's actions. | No | Yes | ### Vercel | Action | What it does | Always a person | Undo | | --- | --- | --- | --- | | Vercel instant rollback | Rolls production back to an earlier deployment. | No | Yes | | Vercel promotion | Promotes a deployment to production. | No | Yes | | Vercel redeploy | Rebuilds and redeploys a deployment. | No | Yes | | Vercel deployment cancel | Cancels a deployment that's still building. | No | No | | Vercel rolling release stage approval | Moves a rolling release to its next stage. | Yes | No | | Vercel rolling release abort | Aborts a rolling release. | No | No | | Vercel Attack Challenge Mode | Turns on Attack Challenge Mode for 15–1,440 minutes. | Yes | Yes | | Vercel firewall IP block | Blocks an IP address. | Yes | Yes | | Vercel firewall IP unblock | Removes an IP block. | Yes | Yes | | Vercel environment variable change | Sets an environment variable to a new value you give, then redeploys. | Yes | Yes | ### Cloudflare | Action | What it does | Always a person | Undo | | --- | --- | --- | --- | | Cloudflare cache purge | Purges cached content for a zone. | No | No | | Cloudflare Worker rollback | Rolls a Worker back to an earlier version. | No | Yes | | Cloudflare load balancer failover | Turns a load balancer pool off or on to move traffic. | Yes | Yes | | Cloudflare security level | Changes a zone's security level. | Yes | Yes | | Cloudflare WAF rule toggle | Turns a WAF rule on or off. | Yes | Yes | ### Feature flags | Action | What it does | Always a person | Undo | | --- | --- | --- | --- | | LaunchDarkly flag kill switch | Turns a flag off or on. | No | Yes | | LaunchDarkly percent rollout | Changes a flag's rollout percentage in stages. | No | Yes | | LaunchDarkly flag version restore | Restores a flag to its state before a recorded change. | No | Yes | | PostHog flag rollout | Changes a flag's rollout percentage in stages. | No | Yes | | PostHog flag switch | Turns a flag off or on. | No | Yes | ### Background jobs | Action | What it does | Always a person | Undo | | --- | --- | --- | --- | | Trigger.dev run replay | After the fix: replays failed runs in stages (up to 1–2,000 runs). | No | No | | Trigger.dev run cancel | Cancels runs in stages (up to 1–2,000 runs). | Yes | No | | Trigger.dev queue pause | Pauses a task's queue. | Yes | Yes | | Inngest run rerun | After the fix: reruns failed runs in stages (up to 1–500 runs). | No | No | | Inngest function pause | Pauses a function. | Yes | Yes | | Inngest function resume | Resumes a paused function. | Yes | Yes | | Inngest run cancel | Cancels runs in stages (up to 1–500 runs). | Yes | No | ### Databases | Action | What it does | Always a person | Undo | | --- | --- | --- | --- | | PlanetScale schema change | Deploys a schema change through a deploy request. | Yes | Yes | | PlanetScale schema revert | Reverts a deployed schema change. | Yes | No | | PlanetScale backfill | Updates rows in batches (1–1,000 rows per batch, up to 100,000 rows) in stages. | Yes | No | | Supabase pooler pool size | Changes the default pool size (1–3,000 connections). | Yes | Yes | | Supabase read-only mode lift | Lifts read-only mode on a project. | Yes | No | | Supabase disk increase | Grows the project's disk (1–60,000 GB). | Yes | No | | Supabase project restart | Restarts the project. | Yes | No | All PlanetScale and Supabase actions write to a data store. ### LiteLLM | Action | What it does | Always a person | Undo | | --- | --- | --- | --- | | LiteLLM model fallback | Routes a failing model to a fallback. | No | Yes | | LiteLLM budget and limits | Changes budgets and rate limits. | Yes | Yes | ### Observability updates These change only what your team sees, so once you allow them on the connection, the ones that can be undone run on their own during incidents. | Action | What it does | Undo | | --- | --- | --- | | Grafana annotation | Marks the incident on a Grafana dashboard. | Yes | | Grafana annotation removal | Removes that annotation. | No | | Grafana alert silence | Silences a Grafana alert for 5–240 minutes. | Yes | | Grafana silence expiry | Ends a silence early. | No | | Axiom annotation | Marks the incident on an Axiom dataset. | Yes | | Axiom annotation removal | Removes that annotation. | No | | Sentry issue resolve | Resolves a Sentry issue. | Yes | | Sentry issue reopen | Reopens a Sentry issue. | No | | Sentry issue assignment | Assigns a Sentry issue. | Yes | | Datadog monitor downtime | Silences a Datadog monitor for 5–240 minutes. | Yes | | Datadog downtime cancellation | Ends a downtime early. | No | | New Relic muting rule | Mutes a New Relic alert for 5–240 minutes. | Yes | | New Relic muting rule removal | Removes a muting rule. | No | | Honeycomb marker | Marks the incident on a Honeycomb dataset. | Yes | | Honeycomb marker removal | Removes that marker. | No | Staged actions, such as rollouts, replays, cancels and backfills, check the incident's alert between stages. You can change **Wait between stages** (60–3,600 seconds) before you approve. ## Next steps - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): Where actions fit in an incident, from first signal to close. - [Operating rules](https://tryace.dev/docs/concepts/operating-rules): Tell Ace how your team wants fixes done. - [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions): What admins and members can do. - [How memory works](https://tryace.dev/docs/concepts/memory): How verified results shape future fixes. --- Source: https://tryace.dev/docs/concepts/operating-rules # Operating rules > Write plain-language rules that tell Ace how your team wants incidents diagnosed and fixed, and understand exactly where Ace applies them and what they can't do. Source: https://tryace.dev/docs/concepts/operating-rules Operating rules are short instructions your admins write once, such as "Prefer a rollback over a forward fix during business hours." Ace follows them when it diagnoses an incident and when it recommends, prepares or explains a fix. Coding agents read them through the [Ace MCP server](https://tryace.dev/docs/mcp). Rules guide Ace's reasoning and never add authority. Your [action policy](https://tryace.dev/docs/concepts/actions-and-approvals) controls what Ace may change and who approves it, and the policy doesn't read operating rules. ## Write a rule 1. In Ace, open **Settings** > **Actions** > **Operating rules**. 2. Select **Add rule**. 3. Write the rule in the text box. 4. Open **Applies to** under the rule and choose the kinds of work it covers. A new rule applies to **Everything** until you change it. 5. Select **Save rules**. To remove a rule, select **×** next to it, then **Save rules**. Saved rules apply from the next investigation or proposal. Only admins can add, change or remove rules, because they shape every investigation in the workspace. Members see the list read-only, with the note "Only admins can change operating rules." ## Where a rule applies | Applies to | Covers | | --- | --- | | **Everything** | All of Ace's reasoning. A rule for **Everything** always applies. | | **Diagnosis** | How Ace diagnoses an incident. | | **Recommendations** | The fixes Ace recommends. | | **Pull requests** | Changes Ace makes to your repositories. | | **Live operations** | Operations on running systems, such as scaling, restarting or failing over. | A rule can cover several kinds of work and applies whenever any of them matches. ## Limits | Limit | Value | | --- | --- | | Rules per workspace | 20. **Add rule** disappears at 20. | | Length of a rule | 400 characters. A counter appears as you near the limit. | | Line breaks | None. Line breaks become spaces when you save. | | Kinds of work | At least one per rule. | | Scope | Your whole organization. There are no per-workload or per-environment rules. | ## Examples | Rule | Applies to | | --- | --- | | Prefer a rollback over a forward fix during business hours. | Recommendations | | For the payments service, treat any change to the ledger tables as a data issue first and check recent migrations before suggesting code changes. | Diagnosis | | Keep fix pull requests to one service and include a test that reproduces the failure. | Pull requests | | Never recommend restarting the orders database during the EU trading day (08:00–17:00 CET). Scale read replicas instead. | Recommendations, Live operations | | Don't scale checkout below 4 tasks. | Live operations | | Our Sentry alerts on the batch importer are noisy at night. Weigh them against the job's own failure metrics before calling a cause. | Diagnosis | Good rules say what to prefer or avoid. Rules apply to the whole organization, so name the service in the text when a rule is about only one. ## How Ace uses your rules | Where | Rules it uses | What they do | | --- | --- | --- | | Investigations | Diagnosis, Recommendations, Pull requests, Live operations and Everything | Shape the root cause analysis and the recommended steps. | | Operations on running systems, including Kubernetes | Recommendations, Live operations and Everything | Shape which operations Ace proposes and how. Approvers see the relevant rules on the plan. | | Fix pull requests | Pull requests and Everything | Narrow the change. They never widen the approved repository or what the pull request is for. | | The Ace MCP server | All, or the kind of work the client asks for | Returned by `get_operating_rules`. | ## What rules can't do Rules can narrow what Ace recommends, set an order of preference or add conditions, but Ace ignores any part of a rule that asks for more authority. A rule can't: - grant a permission, or let someone approve an action they otherwise couldn't - approve or run an action - widen the repositories, resources or operations an action may touch - turn on **Automatic** for an action, or lower the track record it needs - change how confident Ace has to be before it acts Your [action policy and approval rules](https://tryace.dev/docs/concepts/actions-and-approvals) control those. A rule saying "always roll back checkout automatically" has no effect on whether a rollback runs automatically. Rules are guidance, so a recommendation can still miss one. Put anything that must never happen without a person in your action policy instead, by setting the action to **Off** or **Approval**, adding a change freeze or requiring two approvers, and review proposals as usual before you approve them. ## Reading rules from your coding agent MCP clients read the same rules with `get_operating_rules`. To filter by kind of work, pass `applies_to` as `all`, `diagnosis`, `remediation`, `code_change` or `live_operation`, and rules for **Everything** are always included. ```json title="Arguments" { "applies_to": "code_change" } ``` The result lists each rule with the kinds of work it covers, under the note "These narrow what Ace recommends. They never grant permissions or approve anything." ```text Before you open a pull request for INS-1018-ab12, call get_operating_rules with applies_to "code_change" and follow every rule it returns. ``` ## Operating rules and memory Operating rules are what your admins tell Ace, and memory is what Ace learns from your incidents. Neither can expand what Ace may do. For a side-by-side comparison, see [Memory and operating rules](https://tryace.dev/docs/concepts/memory#memory-and-operating-rules). ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): Set what Ace may change and who approves it. - [How memory works](https://tryace.dev/docs/concepts/memory): What Ace learns from incidents, alongside your rules. - [Ace MCP server](https://tryace.dev/docs/mcp): Give coding agents your rules and incident context. --- Source: https://tryace.dev/docs/integrations # Integrations > Every tool Ace connects to, what Ace reads and can change in each, which ones open incidents, and how connecting a tool works. Source: https://tryace.dev/docs/integrations Each integration has its own credential and permissions, and investigations only ever read. Ace changes something in a tool only when the credential allows it and an admin has turned on actions for that integration. ## How integrations work Admins connect integrations under **Settings** > **Data sources**, except Slack, which lives under **Settings** > **Slack**. Most tools take an API key or token. AWS uses a guided setup that creates a read-only IAM role, Kubernetes runs a connector as Deployments in your cluster, and GitHub and Slack use the Ace apps. A workload is something you run, such as an API, a checkout flow or a background worker. An integration's alerts open incidents only on workloads that route it, so in the workload's settings, select the integration under **Investigation triggers** > **Integration alerts**. Many tools send alerts by webhook. The integration's page in Ace shows the URL, and for some tools Ace adds the webhook itself. Ace polls some other tools every 2 minutes. **Investigation sources** in the workload's settings sets which connected sources its investigations read. Once you save the list, Ace reads only those sources plus any source that routes the workload's alerts, so add any source you connect later. Until you save a list, the workload's alert sources decide what Ace reads. Code comes from the GitHub repositories linked to the workload. Integrations that support actions show **Allow Ace to run approved actions**, and monitoring tools show **Allow Ace to update this tool during incidents** instead. Until an admin turns it on, Ace changes nothing in that tool. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). **Disconnect** on an integration's page removes the connection, and where it can, Ace also removes what it set up in the tool, such as its webhook. Afterwards, revoke the credential in the tool itself, using the steps on each integration's page. ## Cloud and infrastructure | Integration | Opens incidents | Ace can change | | --- | --- | --- | | [AWS](https://tryace.dev/docs/integrations/aws) | Yes, from CloudWatch alarms | Nothing with the read role. With the optional actions role, the services you choose, after approval | | [Kubernetes](https://tryace.dev/docs/integrations/kubernetes) | Yes, from failing pods, rollouts and nodes | Nothing with the read connector. With a separate write connector, rollouts, scaling and more, after approval | | [Cloudflare](https://tryace.dev/docs/integrations/cloudflare) | Yes, from Cloudflare notifications | Cache purge, Worker rollback, load balancer failover, security level, WAF rules, after approval | ## Code and deploys | Integration | Opens incidents | Ace can change | | --- | --- | --- | | [GitHub](https://tryace.dev/docs/integrations/github) | No | Fix and revert pull requests, and your deploy workflow to roll back or redeploy | | [Vercel](https://tryace.dev/docs/integrations/vercel) | Yes, from Vercel alerts. Deploys also join the change timeline | Rollback, promotion, redeploy and more, once actions are allowed | ## Collaboration | Integration | Opens incidents | Ace can change | | --- | --- | --- | | [Slack](https://tryace.dev/docs/integrations/slack) | Start an investigation with `@Ace investigate ` | Posts in Slack. Nothing else | ## Observability | Integration | Opens incidents | Ace can change | | --- | --- | --- | | [Datadog](https://tryace.dev/docs/integrations/datadog) | Yes, from monitor notifications | A downtime on the paging monitor during an approved fix, if allowed | | [Prometheus](https://tryace.dev/docs/integrations/prometheus) | Yes, from Alertmanager alerts | Nothing (read-only) | | [Grafana](https://tryace.dev/docs/integrations/grafana) | Yes, from Grafana alerts | Annotations, and a silence on the paging rule during an approved fix | | [New Relic](https://tryace.dev/docs/integrations/new-relic) | Yes, from alert issues | A muting rule on the paging condition during an approved fix | | [Honeycomb](https://tryace.dev/docs/integrations/honeycomb) | Yes, from triggers and SLO burn alerts | Markers when an incident opens and resolves | | [Elasticsearch](https://tryace.dev/docs/integrations/elasticsearch) | No | Nothing (read-only) | | [Axiom](https://tryace.dev/docs/integrations/axiom) | Yes, from monitor alerts | Annotations when an incident opens and resolves | | [Logfire](https://tryace.dev/docs/integrations/logfire) | No | Nothing (read-only) | | [Sentry](https://tryace.dev/docs/integrations/sentry) | Yes, from Sentry alerts | Issue status and assignee, if allowed | ## AI applications | Integration | Opens incidents | Ace can change | | --- | --- | --- | | [LangSmith](https://tryace.dev/docs/integrations/langsmith) | No | Nothing (read-only) | | [LiteLLM](https://tryace.dev/docs/integrations/litellm) | No. Read during investigations | Model fallbacks, budgets and rate limits, after approval | ## Jobs and workflows | Integration | Opens incidents | Ace can change | | --- | --- | --- | | [Trigger.dev](https://tryace.dev/docs/integrations/trigger-dev) | Yes, from failed production runs | Replay or cancel runs, pause or resume a queue, once actions are allowed | | [Inngest](https://tryace.dev/docs/integrations/inngest) | Yes, from failed production runs | Rerun or cancel runs, pause or resume a function, once actions are allowed | ## Databases | Integration | Opens incidents | Ace can change | | --- | --- | --- | | [Supabase](https://tryace.dev/docs/integrations/supabase) | Yes, from project health | Pooler size, read-only mode, disk size, restart. A person always approves | | [PlanetScale](https://tryace.dev/docs/integrations/planetscale) | No. Read during investigations | Deploy requests, and bounded backfills with a write password. A person always approves | ## Product and feature flags | Integration | Opens incidents | Ace can change | | --- | --- | --- | | [PostHog](https://tryace.dev/docs/integrations/posthog) | Yes, from insight alerts | Feature flag rollout and on/off, once actions are allowed | | [LaunchDarkly](https://tryace.dev/docs/integrations/launchdarkly) | No. Changes join the change timeline | Flag on/off, rollout, restore an earlier version, once actions are allowed | ## Paging and incident management | Integration | Opens incidents | Ace can change | | --- | --- | --- | | [PagerDuty](https://tryace.dev/docs/integrations/pagerduty) | Joins the open incident. A high-urgency page opens one if none is open | Nothing in your incidents (read-only) | | [incident.io](https://tryace.dev/docs/integrations/incident-io) | Joins the open incident, or opens one if none is open | Nothing (read-only) | ## Don't see your tool? Tell us which tool you use and what Ace would need to read from it: [talk to our team](https://tryace.dev/book-demo). ## Next steps - [Quickstart](https://tryace.dev/docs/quickstart): connect your first integrations and run an investigation. - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): how alerts from these tools become one incident. - [Security and data](https://tryace.dev/docs/security/security-and-data): what each credential can reach. --- Source: https://tryace.dev/docs/integrations/aws # AWS > Connect AWS so Ace can read CloudWatch, CloudTrail, X-Ray and your infrastructure during investigations, open incidents from CloudWatch alarms, scan for savings and, if you allow it, make approved changes. Source: https://tryace.dev/docs/integrations/aws Ace reads AWS to see what is failing, what changed and how your services connect. A guided setup uses CloudFormation to create a read-only IAM role in your account or organization, and write access comes from a separate, optional role so you can grant it later or never. | | | | --- | --- | | Category | Infrastructure, telemetry and change context | | Ace reads | CloudWatch metrics, alarms and logs, CloudTrail event history, AWS Config history, AWS Health, X-Ray traces, infrastructure metadata, cost recommendations | | Ace can change | Nothing with the read role. With the optional actions role, the AWS services you choose, after approval | | Opens incidents | Yes, from CloudWatch alarms | | Setup | Guided setup in Ace that creates a role in your AWS account | ## What Ace reads During an investigation, Ace reads: - CloudWatch metric data for the incident window, alarm state and history, and error logs through CloudWatch Logs Insights. - Mutating API calls from your existing 90-day CloudTrail event history. Ace never creates a trail or CloudTrail Lake. - Auto Scaling activity, AWS Config resource history and AWS Health events. Health events need an AWS Support plan that includes the AWS Health API, and without one Ace skips them. - The X-Ray service graph and a bounded sample of traces from the incident window. - RDS Performance Insights load by tokenized SQL and wait events, and Application Signals SLO budgets. - Infrastructure metadata for the System Map: networking, compute, databases, queues, storage metadata, tags, quotas and declared relationships. The read role explicitly denies customer data, so it can't read S3 objects, DynamoDB rows, SQS messages, secrets, parameter values, source packages or CloudFormation templates, and it can't decrypt anything or read full SQL text. Ace also strips environment values, inline credentials and secret-bearing command arguments from AWS responses before it stores or shows them. Once the **Cost & performance** panel shows **Verified**, Ace scans daily for savings in Cost Optimization Hub and Compute Optimizer and reads Cost Explorer anomalies. Because Cost Explorer bills per request, Ace makes only one anomaly request per daily scan. ## What Ace can change The read role can't create, update or delete any AWS resource. To let Ace make changes, add the separate actions role with at least one of these groups: | Group | Services | | --- | --- | | Compute | ECS, Auto Scaling, EC2, Lambda, EKS | | Data | RDS, DynamoDB, ElastiCache, SQS, Kinesis | | Network and edge | Load balancers, Route 53, CloudFront, WAF | | Deployments | CodeDeploy, CloudFormation, AppConfig | | Runbooks | SSM documents you allow | | Alarm muting | Mute a CloudWatch alarm during a fix | A change has to clear three checks: 1. The role has permissions only for the groups you chose. 2. Ace proposes changes only on the AWS resources you confirm in each workload's settings. It checks that list again before each write. 3. A person approves the exact plan, and Ace reads the resource state again before it writes. AWS changes need approval unless an admin sets that action to **Automatic** in **Settings** > **Actions** and turns on **Run despite preflight cautions**, which is how Earned autonomy extends to AWS. Some operations always need a person whatever the policy, including stopping an EC2 instance, changing an instance type, RDS instance class or DynamoDB capacity mode, and running an SSM document. Many operations, such as scaling, rollbacks and alarm muting, have an **Undo**. Others, such as a reboot or a forced ECS deployment, can't be undone. Alarm muting runs on its own. When an approved fix starts, Ace mutes the incident's firing CloudWatch alarms on the workload's confirmed resources so on-call isn't paged during the rollout. It mutes an alarm only if every one of its actions is an SNS notification, which means an alarm that also triggers Auto Scaling, EC2, Systems Manager or Lambda actions stays as it is. Ace restores each mute after the fix rolls out or the incident resolves, and anyone can undo a mute from the incident. Review the compute and data groups carefully before you turn them on. Compute lets Ace register task definitions and pass roles, so it can reach as far as the most powerful role in the account that ECS can use, and data can change your data stores. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). ## What opens an incident Ace polls CloudWatch alarm history with the read role, so you don't need SNS or a webhook. - An alarm that moves to `ALARM` opens an incident or joins an open one. - The incident resolves when the alarm returns to `OK`, or moves to `INSUFFICIENT_DATA` after `ALARM`. - When a composite alarm fires, Ace follows the composite and ignores the alarms it covers. An alarm opens an incident only for a workload that routes AWS. To route it, open the workload's settings, turn on **Integration alerts** under **Investigation triggers** and select AWS. **Watch CloudWatch metrics** on the AWS page is a separate setting, off by default because its reads can add CloudWatch charges to your AWS bill. When an admin turns it on, Ace learns each metric's normal range and opens an incident when a deviation lasts. See [Proactive detection](https://tryace.dev/docs/concepts/proactive-detection). ## Before you start - The **admin** role in Ace. See [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions). - Permission in AWS to create CloudFormation stacks and IAM roles. - To connect an AWS Organization: - Organizations must use all features. - A management account administrator must activate StackSets trusted access. In CloudFormation, open **StackSets** and choose **Activate trusted access**. - To launch from a delegated administrator account, first register it as a delegated administrator for CloudFormation StackSets. - Cost scans run from the account you launch from. To cover the whole organization, launch from the management account or the delegated administrator. - For savings recommendations, enroll in Cost Optimization Hub and Compute Optimizer in the launch account. Cost anomalies, Performance Insights and SLOs work without enrollment. ## Credentials and permissions Ace never stores AWS access keys. It assumes the roles the stack creates, using an external ID generated for your workspace, and you can review everything the stack adds in Ace and in the AWS console before you launch it. | Role | Where it lives | What it allows | Why Ace needs it | | --- | --- | --- | --- | | Read role | Each connected account | Read-only access to network, compute, data metadata, application integration, observability, identity metadata, and cost and performance. An explicit deny blocks customer data | Investigations, the System Map, CloudWatch alarms and cost scans | | Organization discovery role | The launch account, in organization mode only | Lists the organization's accounts and organizational units, and assumes the read role in member accounts | Finds member accounts so Ace can read them | | Actions role | Only if you choose action groups | Permissions for the groups you choose. Keeps the read role's data deny | Approved changes and alarm muting | The actions role has its own external ID, so adding or removing it never changes read access. ## Set up ### Connect the read role 1. In Ace, go to **Settings** > **Data sources** > **AWS**. Under **Configuration**, choose an **AWS topology**: - **AWS Organization (recommended)** covers the accounts you choose with one setup. - **Single AWS account** covers one account. 2. Choose a **Primary Region**. 3. For an organization, enter the **Root or OU IDs**. A root ID covers every account, and OU IDs cover only those OUs. Choose the **Launch account**, then confirm that StackSets trusted access is activated. 4. Optional: under **Actions (optional)**, choose the groups Ace may change. Leave them all off for read-only access, since you can add actions later. 5. Choose **Launch stack in AWS**, then review the stack in the AWS console and create it. In organization mode, the stack deploys the read role to every account in the chosen roots or OUs, including accounts added later. 6. When the stack completes, open its **Outputs** tab and copy **PrimaryRoleArn**. Paste it into **Organization discovery role ARN**, or **Member role ARN** for a single account, and choose **Connect**. Ace validates the role, then discovers your infrastructure for the System Map. ### Add actions later 1. In **Settings** > **Data sources** > **AWS**, find **AWS actions**. 2. Under **Choose what Ace can change**, select the groups. In organization mode, also choose the **Organizational units that receive the role**. 3. Choose **Launch stack** and create it in the AWS console. 4. When the stack completes, choose **Verify and turn on**. 5. Confirm the AWS resources in each workload's settings. ### Turn on cost scans 1. Check the status in the **Cost & performance** panel. 2. If it shows **Not enrolled**, enroll in Cost Optimization Hub and Compute Optimizer. In an organization, enroll from the launch account and include member accounts. 3. Recommendations appear within a day, and then you can choose **Re-check**. If the panel shows **Update available**, update the stack in the AWS console as Ace shows, then choose **Re-check**. ## Confirm it's working - The AWS panel shows **Connected** with **AWS Organization · member accounts discovered automatically**, or **Standalone account** and your Regions. - The role access and CloudTrail access checks pass. If one fails, fix the cause and choose **Re-validate**. - The System Map fills in after discovery. - The **Cost & performance** panel shows **Verified**. - With actions on, **AWS actions** shows **On** with the account, verification time and groups. **Confirmed resources** lists each workload. ## Disconnect To turn off actions only, choose **Turn off** in **AWS actions**. Ace stops making AWS changes and plans waiting for approval won't run. If you turn actions back on later, you confirm each workload's resources again, and the stack stays in your account until you delete it. To disconnect AWS, choose **Disconnect** in **Settings** > **Data sources** > **AWS**. Ace stops reading AWS and cancels pending AWS plans, but it doesn't delete anything in AWS. To remove Ace's access, delete the Ace stacks in CloudFormation: - The actions stack, if you launched one separately. - The main Ace stack, from the account you launched it from. In organization mode, this also removes the read role from member accounts. ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): approval, automatic policies and undo. - [Proactive detection](https://tryace.dev/docs/concepts/proactive-detection): CloudWatch watches and early warnings. - [Security and data](https://tryace.dev/docs/security/security-and-data): what Ace stores and how it is protected. --- Source: https://tryace.dev/docs/integrations/kubernetes # Kubernetes > Connect a Kubernetes cluster so Ace can read resources, events and logs in the namespaces you choose, open incidents from failing pods and nodes and, if you allow it, make approved changes. Source: https://tryace.dev/docs/integrations/kubernetes Ace reads pods, rollouts, events and logs from your cluster during an investigation. The connector runs as Deployments in your cluster, with a read-only reader and an optional, separate writer for approved changes. Both connect out to Ace over HTTPS, so you don't open an inbound port, expose the Kubernetes API or upload a kubeconfig, and Kubernetes credentials stay in the cluster. | | | | --- | --- | | Category | Cluster resources, events, logs and governed remediation | | Ace reads | Resources, events and container logs in the namespaces you choose. Optionally nodes and other cluster-wide resources | | Ace can change | Nothing with the read connector. With a separate write connector, rollouts, scaling, resources and more, after approval | | Opens incidents | Yes, from failing pods, rollouts and nodes | | Setup | Connector Deployments in your cluster, installed with commands Ace generates | ## What Ace reads In each namespace you choose, the reader can `get` and `list` these resources and read current and previous container logs: - Pods, Services, Endpoints, EndpointSlices, ConfigMaps, Events, PersistentVolumeClaims, ResourceQuotas, LimitRanges, ServiceAccounts - Deployments, ReplicaSets, StatefulSets, DaemonSets, ControllerRevisions, Jobs, CronJobs - Ingresses, NetworkPolicies, HorizontalPodAutoscalers, PodDisruptionBudgets, Roles, RoleBindings - Argo CD Applications, Argo Rollouts, Flux Kustomizations and HelmReleases, when your cluster serves them The reader never reads or writes Secrets. **Read cluster-wide resources such as nodes and admission webhooks** adds Nodes, Namespaces, PersistentVolumes, StorageClasses, ClusterRoles, ClusterRoleBindings, and mutating and validating webhook configurations. Without it, the only thing the reader reads outside your chosen namespaces is the `kube-system` namespace, which it uses to identify the cluster. An investigation reads only the namespaces mapped to the incident's workload. ## What Ace can change The reader can't change anything. For changes, install a separate write connector and confirm which namespaces it can act on for each workload and environment, and Ace can then propose: - Rollouts: restart, undo, pause and resume, and set a container image. - Capacity: scale, update an autoscaler, set container resources, update a disruption budget. - Jobs: suspend a CronJob. Resume a CronJob or re-run a Job (approval only). - Configuration: set a non-secret environment value (approval only). - Pods and nodes: delete a pod, and cordon, uncordon, taint, untaint or drain a node. These are all approval only, and node operations need cluster administration. - GitOps controllers: Argo Rollouts abort, undo, promote and retry, Argo CD rollback, sync and auto-sync on or off, and Flux reconcile, suspend and resume. - An exact patch, create or delete on a supported resource. A person approves each exact operation unless an admin grants automatic repairs, and approval-only operations never run automatically. Most operations have an **Undo**, including scale, rollout undo, and image and resource changes. A rollout restart, pod delete, Job re-run and node drain can't be undone, and undoing a drain only uncordons the node. Automatic repairs are how Earned autonomy works in Kubernetes, and they need their own grant. An admin names the exact resource, field and allowed value, sets an expiry of at most seven days, a daily limit and a recovery window, and then confirms **I authorize automatic restoration of these fields and values for this workload and environment.** Ace repairs automatically only within that grant and only after it delivers a countdown to Slack, so your team sees the repair coming. If a repair makes things worse, automatic repairs stop. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). ## What opens an incident The reader watches pod, rollout and node state in the mapped namespaces: | Condition | Severity | | --- | --- | | `CrashLoopBackOff` | Critical | | `ImagePullBackOff` or `ErrImagePull` | Critical | | Deployment `ProgressDeadlineExceeded` | Critical | | Node `NotReady` | Critical | | Repeated `OOMKilled` restarts | High | | Recent pod eviction | Medium | | Node `MemoryPressure`, `DiskPressure` or `PIDPressure` | Medium | Node conditions need cluster-wide read access, because nodes sit outside any namespace. A condition opens an incident only for a workload that is mapped to that namespace and routes the cluster. To route it, open the workload's settings, turn on **Integration alerts** under **Investigation triggers** and select the cluster. The connector doesn't monitor application health, so for that, add an HTTP health check or connect an alert source. See [Proactive detection](https://tryace.dev/docs/concepts/proactive-detection). ## Before you start - The **admin** role in Ace. - A Kubernetes account that can create a namespace, a Secret and RBAC bindings. - Outbound HTTPS from the cluster to Ace, and access to the connector's image registry (`ghcr.io`, or your mirror). - Helm 3.8 or later and `kubectl`, or the setup manifest if you don't use Helm. - 1 to 50 existing namespaces per connection. ### Private networks The connector works with private clusters, including EKS, GKE and AKS with a private API endpoint, and on-premises clusters. The reader and writer call the Kubernetes API from inside the cluster, so the endpoint doesn't need to be public. Ace never connects in, so you don't open inbound ports or allowlist Ace. Allow outbound HTTPS from the connector pods in `ace-connectors` to the Ace address in the install commands, and make sure your nodes can pull the image from `ghcr.io` or your mirror. ## Credentials and permissions Ace generates a credential for each connector, which you download once. The install commands store it as a Secret in the `ace-connectors` namespace. | Connector | Kubernetes permissions | Why Ace needs them | | --- | --- | --- | | Read, in each chosen namespace | Role with `get` and `list` on the resources listed above (no Secrets), plus `pods/log` | Investigations, discovery and incident detection | | Read, cluster identity | ClusterRole with `get` on the `kube-system` namespace only | Identifies the cluster | | Read, optional cluster-wide | ClusterRole with `get` and `list` on nodes, namespaces, persistent volumes, storage classes, cluster roles and bindings, and admission webhook configurations | Node conditions and cluster-wide resources | | Write, in each chosen namespace | Role with `get`, `list`, `patch`, `create` and `delete` on all resources | Approved changes | | Write, optional **Grant cluster administration** | ClusterRole with all verbs on all resources | Node operations and other cluster-wide changes | > **Write access reaches data** > > Namespace write access can create workloads that read data and credentials in those namespaces. Ace still acts only within a confirmed action scope and your action policy, but review the permissions before you install a writer. The connector runs as a non-root user with a read-only root filesystem, all Linux capabilities dropped and the `RuntimeDefault` seccomp profile. ## Set up ### Connect the read connector 1. In Ace, go to **Settings** > **Data sources** > **Kubernetes**. 2. Under **Connect a cluster**, enter: - **Cluster name**, unique in your workspace, for example `production-eu`. - **Namespaces**: comma-separated existing namespaces. Each workload can investigate only its mapped namespaces. - **Workload**: the workload to map, or **Map a workload later**. - Optional: **Read cluster-wide resources such as nodes and admission webhooks**. 3. Choose **Create connection**. 4. Under **Download the connector credential**, choose **Download credential file**. The file is available only during setup, so keep it until the install finishes. 5. Under **Install with Helm**, enter your **Kubernetes context**: ```bash kubectl config get-contexts ``` 6. Choose **Copy Helm commands** and run them from the folder that holds the credential file. They create the `ace-connectors` namespace and the credential Secret, delete the file, install the reader and wait for it to start. The credential never appears on the command line. 7. Under **Check the connection**, choose **Verify connection**. When it shows **Connection verified**, choose **Finish setup**. 8. Under **Workload mapping**, pick a workload and its namespaces, and choose **Save mapping**. For a mirror or private registry, expand **Use a registry mirror or private registry?** before you copy the commands, then enter the mirrored **Connector image** by digest and, optionally, an **Image pull secret** that already exists in `ace-connectors`. With Argo CD or Flux, use the same chart and values, and provide the credential Secret through your secret manager. Without Helm, expand **Install with a manifest instead** and choose **Download setup manifest**. It creates the same objects and also contains the credential, so store it securely. To connect more clusters, choose **Add another cluster**. ### Allow approved changes (optional) 1. On a verified cluster, choose **Action access**. 2. Optional: turn on **Grant cluster administration**. 3. Choose **Create write connector** and install it like the reader, with the commands Ace generates. It runs as a separate Deployment with its own credential. 4. Under **Workload action scope**, select a mapped workload, its **Environment** and its **Namespaces in this environment**. Each namespace belongs to one environment. Optionally turn on **Permit cluster administration for this workload** or **Permit approved container commands**. 5. Choose **Verify and confirm action scope**. A writer on its own doesn't turn on automatic repairs. ## Confirm it's working - The install panel shows **Connection verified** with each check. If it shows **Connection check is pending. Run it again shortly.**, wait for the connector to start and retry. - If validation times out, check that the connector pod is running, then check DNS, outbound HTTPS and certificate trust from the cluster to Ace. The connector's logs leave out credentials and payloads. - The cluster card shows the last connection check and its namespaces. **Validate and discover** runs the checks again, and discovery checks **Cluster identity** and **Scoped inventory**. - If the card asks you to upgrade the connector, follow the steps shown and upgrade the reader and writer together. ## Disconnect In Ace: - **Rotate credential** revokes the current credential, so run the install commands again with the new file. - **Revoke write access** removes the writer's access. **Replace write credential and grant** revokes the old writer credential and invalidates every action scope. - **Disconnect** deletes the connection. In the cluster, uninstall the reader and writer with `helm uninstall`, or delete the objects from the setup manifest. Then delete their credential Secrets in `ace-connectors`, using the names from your install commands, and keep the `ace-connectors` namespace if other connectors still run in it. ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): approval, automatic policies and undo. - [Prometheus](https://tryace.dev/docs/integrations/prometheus): add metrics and Alertmanager alerts. - [Slack](https://tryace.dev/docs/integrations/slack): needed for automatic repair countdowns. --- Source: https://tryace.dev/docs/integrations/github # GitHub > Install the Ace GitHub App so Ace can read recent code changes, deploys and CI runs during investigations and, after approval, open fix and revert pull requests or roll back a deploy. Source: https://tryace.dev/docs/integrations/github Ace uses GitHub to tie an incident to the code change or deploy that caused it, and you connect it by installing the Ace GitHub App on the repositories you choose. | | | | --- | --- | | Category | Recent code changes, deploys and CI context | | Ace reads | Commits, diffs, pull requests, file contents, code search, workflow runs, job logs, check annotations, deployments | | Ace can change | Opens fix and revert pull requests, and runs your deploy workflow to roll back or redeploy, after approval or under your action policy | | Opens incidents | No | | Setup | GitHub App | ## What Ace reads During an investigation, Ace reads these from the repositories mapped to the incident's workload: - Commits, commit patches, comparisons between revisions, and the pull requests for a commit. - File contents and code search results. - GitHub Actions workflow runs, jobs, the tail of job logs and check annotations. - Deployments and their status. Ace records each deployment status on a mapped repository as a change on that workload, and a successful deploy updates the workload's open incidents. ## What Ace can change | Action | What it does | Can be undone | | --- | --- | --- | | Fix pull request | Opens a pull request with a code or configuration fix | Yes | | Revert pull request | Opens a pull request that reverts a change | Yes | | Roll back deploy | Runs your deploy workflow at an earlier revision | Yes | | Redeploy | Runs your deploy workflow again at the current revision, with no code change | No | Ace proposes these on the incident, and each one runs only after a person approves the exact plan or when your action policy lets it run automatically for that workload. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). Your team merges the pull requests a person approved. Ace merges its own pull request only when the action ran under an automatic policy with **Auto-merge pull requests** on, every Ace reviewer passed, checks are green and Ace announced the merge in Slack first, which gives anyone the chance to stop it. **Roll back deploy** and **Redeploy** need a deploy workflow for the workload, which you connect under **Rollback workflow** in **Settings** > **Actions**. The repository must be mapped to the workload, and the workflow must be active and deploy to an existing GitHub environment. ## What opens an incident GitHub doesn't open incidents. CI failures and deploys add evidence to incidents that other sources open. ## Before you start - The **admin** role in Ace. - Permission to install GitHub Apps on the account or organization, which means an organization owner, or a repository admin if your organization allows it. - One GitHub App installation per Ace workspace. ## Credentials and permissions Ace uses a GitHub App and never a personal access token or an OAuth user token, so access doesn't depend on any one person's account. You choose which repositories the app can see, and GitHub lists its permissions on the install page before you approve. The Ace GitHub App gets read-only access to the repositories you choose, and Ace opens pull requests and runs your deploy workflow only for the approved actions above. ## Set up 1. In Ace, go to **Settings** > **Data sources** > **GitHub**. 2. Under **Install the Ace GitHub App**, choose **Install GitHub App**. GitHub opens in a new tab. 3. Choose the account or organization, then **All repositories** or **Only select repositories**. 4. Review the permissions and choose **Install**. GitHub returns you to Ace, which shows **GitHub App connected**. 5. In each workload's settings, map its repositories under **Code context**. Only workspace admins can change these mappings. 6. Optional: for each workload you want Ace to roll back, set up a **Rollback workflow** in **Settings** > **Actions**. ## Confirm it's working - The GitHub panel shows the connected account and the number of repositories with access. - **Repositories with access** lists each repository with its visibility and default branch. - If it shows **No repositories with access yet.**, choose **Add repositories on GitHub** and select repositories in the installation settings. ## Disconnect In Ace, choose **Disconnect** in **Settings** > **Data sources** > **GitHub**. Ace deletes the connection and its repository mappings, but the app stays installed on GitHub. To change access in GitHub, choose **Manage on GitHub** in Ace, or open your account or organization settings > **GitHub Apps**, and then do one of these: - **Uninstall** the app to remove all access. Ace deletes the connection and mappings when GitHub reports it. - **Suspend** the app to pause access. Ace marks the connection suspended until you unsuspend it. - Remove single repositories from the installation. Ace drops them and their mappings. ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): approval, automatic policies and undo. - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): how code fits into an investigation. - [Ace MCP server](https://tryace.dev/docs/mcp): give your coding agent the same incident details. --- Source: https://tryace.dev/docs/integrations/slack # Slack > Install the Ace Slack app so your team can start investigations with @Ace, follow incidents in your channels and approve Ace's actions in Slack. Source: https://tryace.dev/docs/integrations/slack Ace posts incidents and weekly reports to the Slack channels you choose, answers when someone mentions it, and posts action plans that your team can approve right in the thread. | | | | --- | --- | | Category | Communication | | Ace reads | Messages that mention Ace, direct messages to Ace, replies in threads Ace takes part in, reactions, channel list and member directory | | Ace can change | Posts messages, reactions and files in Slack. Nothing else | | Opens incidents | You can start an investigation with `@Ace investigate ` | | Setup | Slack app | ## What Ace reads - Messages that mention `@Ace`, and direct messages to Ace. - Replies in threads Ace takes part in, including edits and deletions of replies to Ace. - Reactions and button clicks on Ace's messages. - Your channel list, so admins can choose where Ace posts. - Your member directory, so admins can link Ace members to their Slack accounts. Ace redacts credentials and personal details from a Slack reply before it stores the reply as an engineer note or in memory. See [Memory](https://tryace.dev/docs/concepts/memory) and [Security and data](https://tryace.dev/docs/security/security-and-data). ## What Ace can change Ace changes nothing outside Slack, and it posts to the channel you choose for each purpose: | Purpose | What Ace posts | | --- | --- | | **Insights** | Real-time incidents and investigation starts | | **Weekly reports** | Incident summaries and organization-wide analysis | Ace also replies in threads and direct messages, adds reactions and uploads files. With **Show each teammate in Slack** on, each of Ace's teammates posts under its own name and icon, and with it off, every update comes from Ace. ### Approve actions in Slack When Ace proposes an action, it posts the exact plan in the incident thread. The card has **Decline** and an approve button named for the action, such as **Approve rollback** or **Generate fix**. It also shows **Edit values** when the plan has values you can change, and **Something else…** to ask for a different plan in your own words. Because a click in Slack only tells Ace which Slack user clicked, Ace checks the person who clicked against your Ace roles: - They must be an Ace member with a linked Slack account. - A production change needs an admin, or a member with approval permission for that kind of action and environment. - If someone without permission tries to approve, Ace mentions up to three people in the thread who can. Under Earned autonomy, an automatic action still gives your team a chance to stop it. Before one runs, Ace posts a countdown in Slack with a way to cancel, and if Ace can't deliver the countdown, the action doesn't run. Automatic Kubernetes repairs and automatic pull request merges both depend on it. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). You can also ask for a change in plain words, for example `@Ace scale checkout to 6 tasks`, and Ace prepares the exact plan and posts it for approval. ## What opens an incident Slack isn't an alert source, but a linked Ace member can start an investigation: ```text @Ace investigate checkout-api ``` Ace finds the workload by name and replies with a confirmation only you can see, with **Start investigation** and **Cancel**. If the name matches more than one workload, Ace asks which one you mean. When you confirm, Ace investigates the workload over the last hour and links to the incident. Ace also understands these commands: | Command | What it does | | --- | --- | | `help ` | Lists what Ace can do on that workload | | `runbook` | Shows open actions | | `runbook ` | Shows actions for one incident | | `stop` | Ace leaves the thread | ## Before you start - You're signed in to Ace in a workspace that has finished onboarding. - You can install apps in your Slack workspace. - Only an Ace **admin** can choose channels and link members' Slack accounts. ## Credentials and permissions Ace connects through Slack OAuth with these bot scopes: | Scope | Why Ace needs it | | --- | --- | | `app_mentions:read` | Receives messages that mention `@Ace` | | `channels:history`, `groups:history`, `im:history`, `mpim:history` | Reads replies in threads Ace takes part in, in public and private channels and direct messages | | `channels:read`, `groups:read`, `im:read`, `mpim:read` | Lists channels and conversations for routing | | `chat:write` | Posts incidents, reports, replies and approval cards | | `chat:write.customize` | Posts under a teammate's name and icon | | `files:write` | Uploads files to threads | | `reactions:read`, `reactions:write` | Reads reactions on Ace's messages and adds its own | | `users:read` | Lists members so admins can link them to Ace | | `assistant:write` | Shows Ace's status in Slack's agent view | ## Set up 1. In Ace, go to **Settings** > **Slack** and choose **Connect Slack**. Slack opens in a popup, so if your browser blocks it, allow popups for Ace and try again. 2. In Slack, choose your workspace, review the permissions and choose **Allow** within 15 minutes. 3. Under **Channels**, use **Add a channel…** to add a channel to **Insights** and to **Weekly reports**. 4. In **Settings** > **Team**, link each member's **Slack account** with **Choose a Slack account**. Members need a linked account to approve actions or start investigations in Slack, and the person who installed the app is linked automatically. ## Confirm it's working - **Settings** > **Slack** shows **Connected** with your Slack workspace name and **Ace is live in this workspace**. - When you mention `@Ace` in a channel, Ace replies in the thread. - If the connection fails, Ace shows why, for example **Slack authorization was denied or cancelled.** or **Authorization request expired. Please try again.** ## Disconnect Ace has no disconnect button for Slack. Instead, remove the Ace app from your workspace in Slack's app management, which revokes Ace's bot token. To unlink one member, unlink their Slack account in **Settings** > **Team**. ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): who can approve and how Ace announces automatic actions. - [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions): admins, members and approval permission. - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): from the first alert to a closed incident. --- Source: https://tryace.dev/docs/integrations/datadog # Datadog > Connect Datadog so Ace can read metrics, logs, APM traces and monitors during investigations, open incidents from monitor alerts and mute the paging monitor while an approved fix rolls out. Source: https://tryace.dev/docs/integrations/datadog Ace reads Datadog metrics, logs, traces and monitors during investigations, and monitor notifications open incidents. If you allow it, Ace also puts the paging monitor into a downtime while an approved fix rolls out. | | | | --- | --- | | Category | APM, metrics and monitors | | Ace reads | Metrics, monitors, error logs, APM spans and traces, the service map, deployment versions, Error Tracking issues, usage | | Ace can change | Monitor downtimes during an approved fix, if you allow it. Alert wiring adds a webhook and edits monitor messages | | Opens incidents | Yes, from monitor notifications | | Setup | Settings form | ## What Ace reads During an investigation, Ace reads: - Metric time series for the incident window, and monitor status. - Recent error logs. - APM spans and aggregates (counts, failures, p95 and p99 latency), full traces, the service map, and deployment versions from the `version` tag. - Error Tracking issues. When you connect, Ace discovers your active metrics and monitors, plus the services and environments in monitor tags. With the optional `usage_read` scope, it also scans for custom metrics nobody queries. ## What Ace can change If the application key has `create_webhooks` and `monitors_write`, Ace creates its own webhook and mentions it in the message of every monitor that can open incidents, as listed below. Ace keeps those mentions current, so new monitors get one and monitors that can't open incidents lose it. If an admin allows it, Ace puts the paging monitor into a downtime when an approved fix starts, so on-call isn't paged during the rollout. The downtime covers only the alerting group and ends on its own. Downtimes run without approval unless an admin requires one in an action policy, and you can undo each one from the incident. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). ## What opens an incident A monitor notification opens an incident, or joins an open one, when its transition is `Triggered`, `Re-Triggered`, `Warn`, `Re-Warn`, `No Data`, `Re-No Data` or `Renotify`, and `Recovered` resolves it. Ace groups notifications by monitor and scope. | Notification | Severity | | --- | --- | | `No Data` | Low | | `Warn` | Medium | | Priority P1 | Critical | | Priority P2 | High | | Priority P3 | Medium | | Priority P4 or P5 | Low | | No priority | High | These monitors never open incidents: - CI pipeline and CI test monitors. - Cloud Cost and Audit Trail monitors. - Synthetic tests tagged `ci`, `ci:true`, `ci_only`, `ci-only`, `env:ci` or `source:ci`. - Any monitor tagged `ace:ignore`. A notification opens an incident only for a workload that routes Datadog. To route it, open the workload's settings, turn on **Integration alerts** under **Investigation triggers** and select Datadog. Ace doesn't poll Datadog for alerts, so until the webhook delivers, Ace uses Datadog only as evidence. ## Before you start - The **admin** role in Ace. - A Datadog API key and application key on the US1, US3, US5, EU, AP1, AP2 or US1-FED site. Ace calls Datadog's public API for that site, with no custom endpoint or proxy option. ## Credentials and permissions | Credential | Where to create it | Scopes | Why Ace needs it | | --- | --- | --- | --- | | API key | **Organization Settings** > **API Keys** > **New Key** | None | Authenticates API calls | | Application key | **Organization Settings** > **Application Keys** > **New Key** | See below | Reads data and, optionally, wires alerts and creates downtimes | Application key scopes: | Scope | Needed for | | --- | --- | | `timeseries_query`, `metrics_read`, `monitors_read`, `logs_read_data` | Required | | `create_webhooks`, `monitors_write` | Alert wiring by Ace | | `monitors_downtime` | Monitor downtimes | | `apm_read`, `error_tracking_read` | APM and Error Tracking evidence | | `usage_read` | Cost scans | Select only these scopes, because an unscoped key carries all of its owner's permissions. You can also create the key on a service account with the **Datadog Read Only Role**, though alert wiring, downtimes and cost scans still need their extra scopes. ## Set up 1. In Ace, go to **Settings** > **Data sources** > **Datadog**. 2. Under **Credentials**, choose your **Datadog site** and enter the **API key** and **Application key**. **Create key** opens the application keys page for your site. 3. Choose **Connect Datadog**. 4. Under **Alerts**, if the key has `create_webhooks` and `monitors_write`, Ace sets up the webhook and shows **Ace set up a webhook mentioned in every monitor.** 5. Otherwise, set it up by hand: 1. Copy the webhook URL Ace shows. 2. Choose **Generate token** and copy the token. Ace shows it once. 3. In Datadog, open **Integrations** > **Webhooks** and add a webhook with that URL. Set **Custom Headers** to: ```json {"Authorization": "Bearer "} ``` 4. Mention `@webhook-` in the message of each monitor that should open incidents. 6. Optional: turn on **Allow Ace to update this tool during incidents** to let Ace create monitor downtimes during approved fixes. 7. In each workload's settings, select Datadog under **Integration alerts**. You can't edit Datadog credentials in place, so to change keys, disconnect and connect again. ## Confirm it's working - Validation shows **API authentication** passing, and **APM spans**, **Error Tracking** and **Usage metering** either pass or are skipped with the scope to add. - The alerts status moves from **Waiting for first event** to **Verified** on the first signed delivery. To check right away, send a test notification from a monitor. - If it shows **Last delivery failed** because the token didn't match, copy the current token into the webhook's `Authorization` header. - With downtimes allowed, the action list shows **Datadog monitor downtime** as **Available**. If it shows that it needs `monitors_downtime`, add that scope. ## Disconnect In Ace, choose **Disconnect** in **Settings** > **Data sources** > **Datadog**. Ace removes its webhook mention from every monitor, deletes the webhook and then deletes the connection, and anything it can't remove appears as a warning for you to clean up in Datadog. In Datadog, revoke the application key and API key you created for Ace in **Organization Settings**. If you wired alerts by hand, delete the webhook and remove the mention from your monitors. ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): how observability updates and undo work. - [Memory](https://tryace.dev/docs/concepts/memory): how Ace learns from your Datadog alerts. - [Integrations](https://tryace.dev/docs/integrations): connect more data sources. --- Source: https://tryace.dev/docs/integrations/prometheus # Prometheus > Connect Amazon Managed Service for Prometheus or an in-cluster Prometheus on Amazon EKS so Ace can query metrics during investigations and open incidents from Alertmanager alerts. Source: https://tryace.dev/docs/integrations/prometheus Ace queries Prometheus for the affected workload's latency, error rate, throttling, memory and restarts, and with Alertmanager connected, your alerts open incidents. Ace supports two deployment types: - **Amazon Managed Prometheus (AMP)**. Ace queries the AMP API, and alerts arrive through an Amazon SNS topic. - **In-cluster Prometheus (EKS)**. Prometheus Operator or kube-prometheus-stack in an Amazon EKS cluster, which Ace queries through the cluster's Kubernetes API server. Alerts arrive through an Alertmanager webhook. Other self-hosted Prometheus servers aren't supported directly, but you can remote-write their metrics to an AMP workspace and connect that. | | | | --- | --- | | Category | Kubernetes metrics and alerts | | Ace reads | PromQL queries, scrape targets, rules and metric metadata. With alerts on, Alertmanager status | | Ace can change | Nothing (read-only) | | Opens incidents | Yes, from Alertmanager alerts, if you turn on alert triggers | | Setup | Guided setup in Ace that creates a role in your AWS account | ## What Ace reads During an investigation, Ace runs PromQL range queries over the incident window for p95 latency, request rate, error rate, CPU throttling, memory pressure and usage, restart rate and OOM kills. It also reads scrape targets, rules and metric metadata. On AMP, Ace queries your workspace through the AMP API. On EKS, it queries the Prometheus service through the Kubernetes API server's service proxy and can read resources only in the Prometheus namespace you choose. ## What Ace can change Ace changes nothing in Prometheus. Its role can't change rules, scrape configuration or your Alertmanager definition, and on EKS it also can't deploy workloads, edit manifests or change RBAC. ## What opens an incident Only Alertmanager alerts open incidents, and only if you turn on **Alert triggers** during setup. - A firing alert opens an incident or joins an open one. - A resolved alert resolves the matching incident. - Alerts named `Watchdog` or `InfoInhibitor`, and alerts with severity `info` or `none`, never open incidents. An alert opens an incident only for a workload that routes it. To route it, open the workload's settings, turn on **Integration alerts** under **Investigation triggers** and select **Alertmanager via AMP** or **Alertmanager via EKS Prometheus Operator**. Ace doesn't poll Prometheus for alerts, so without Alertmanager alerts, Ace uses Prometheus only as evidence. ## Before you start - The **admin** role in Ace. - Permission in AWS to create CloudFormation stacks and IAM roles. - **AMP**: an AMP workspace and, for alerts, an AMP Alertmanager definition you can edit. The alerts stack creates an SNS topic, billed at standard Amazon SNS pricing. - **EKS**: an EKS cluster whose authentication mode allows access entries, with Prometheus running in one namespace, and, for alerts, access to your Alertmanager configuration. Settings shows one Prometheus connection, either AMP or EKS, so to change its settings, disconnect and connect again. ### Private networks - **AMP**: Ace queries the AMP API with the role the setup creates and alerts arrive through Amazon SNS, so nothing in your VPC needs to be reachable. For a private EKS cluster, remote-write from your in-cluster Prometheus to AMP and connect the workspace. - **EKS**: Ace connects from its cloud to your cluster's Kubernetes API endpoint, so the cluster needs public endpoint access, and a cluster with only private endpoint access must use AMP. For alerts, Alertmanager needs outbound HTTPS to Ace. The [Kubernetes](https://tryace.dev/docs/integrations/kubernetes) connector runs inside the cluster and connects out to Ace, so it works with a private API endpoint. ## Credentials and permissions Ace never stores AWS access keys. The guided setup uses CloudFormation to create an IAM role, which Ace assumes with an external ID generated for your workspace, and you review the stack in the AWS console before you create it. | Deployment type | What the setup creates | Why Ace needs it | | --- | --- | --- | | AMP | A role with read-only query access to your workspace only: metrics, series, labels, metric metadata and rules | Metrics queries and rules | | AMP, alerts on | Read access to Alertmanager status, and an SNS topic that only AMP can publish to, from your workspace. An HTTPS subscription delivers to Ace | Alertmanager status and alert delivery | | EKS | A role that can describe your cluster only, and an EKS access entry with the AWS-managed `AmazonEKSAdminViewPolicy`, scoped to the Prometheus namespace | Read-only access in that namespace, including the service proxy used to query Prometheus and Alertmanager | | EKS, alerts on | A webhook secret generated by Ace. Alertmanager sends it as a bearer token | Authenticates alert deliveries | ## Set up In Ace, go to **Settings** > **Data sources** > **Prometheus** and choose your deployment type. **AMP** 1. Choose **Amazon Managed Prometheus (AMP)**. 2. Under **Capabilities**, **Metrics querying** is always on. Turn on **Alert triggers** if you want Alertmanager alerts to open incidents. 3. Under **Configuration**, choose the **Region** and enter the **AMP Workspace ID** (`ws-...`). 4. Choose **Launch stack in AWS**, then review the stack in the AWS console and create it. 5. When the stack completes, copy the **Role ARN** from its outputs into Ace, and if alerts are on, the **SNS Topic ARN** too. 6. Choose **Connect**. 7. If alerts are on, add Ace to your AMP Alertmanager definition. Ace shows the snippet with your topic ARN and Region: ```yaml title="AMP Alertmanager definition" alertmanager_config: | route: receiver: default routes: - receiver: ace-alerts-sns continue: true receivers: - name: default - name: ace-alerts-sns sns_configs: - topic_arn: "" sigv4: region: "" subject: "AceAlert" ``` In the Amazon Managed Service for Prometheus console, open **All workspaces** > your workspace > **Alert manager** and choose **Modify**, or **Add definition** if the workspace has none. Merge the `ace-alerts-sns` receiver and route into your existing definition, keeping `continue: true` so alerts still flow to your existing receivers. **EKS** 1. Choose **In-cluster Prometheus (EKS)**. 2. Under **Capabilities**, **Metrics querying** is always on. Turn on **Alert triggers** if you want Alertmanager alerts to open incidents. 3. Under **Cluster details**, enter the **Cluster region**, **EKS Cluster Name** and **Prometheus Namespace** (default `monitoring`). 4. Choose **Launch stack in AWS**, then review the stack in the AWS console and create it in the account that owns the cluster. 5. When the stack completes, copy the **Role ARN** from its outputs into Ace. 6. Under **Prometheus service**, enter the **Service Name** (for example `prometheus-operated`) and **Port** (for example `9090`). 7. If alerts are on, choose **Generate secret** under **Webhook Secret** and store it, because you can't see it again after you connect. 8. Choose **Connect**. 9. If alerts are on, add Ace as a receiver in your `alertmanager.yaml` or Alertmanager custom resource. Ace shows the snippet with your webhook URL: ```yaml title="alertmanager.yaml" receivers: - name: ace-alerts webhook_configs: - url: "" send_resolved: true http_config: authorization: type: Bearer credentials: "" route: routes: - receiver: ace-alerts continue: true ``` Copy the webhook URL Ace shows and use the secret from step 7. Keep `continue: true` so alerts still flow to your existing receivers. ## Confirm it's working - The panel shows **Validating integration…** while Ace checks access. On AMP, it checks role access and runs a test query, and on EKS, it checks role access, the cluster, and Prometheus build info, targets and rules. With alerts on, it also checks Alertmanager. - When validation passes, the panel shows **Connected** with the Region. - With alerts on, the panel shows **Alert triggers — no events received yet** until the first alert arrives, then **Alert triggers active**. ## Disconnect In Ace, open the connection in **Settings** > **Data sources** > **Prometheus** and choose **Disconnect**. Then: 1. Remove the `ace-alerts-sns` or `ace-alerts` receiver and route from your Alertmanager configuration. 2. Delete the Ace Prometheus stack in CloudFormation, which removes the IAM role along with the SNS topic and subscription or the EKS access entry. ## Next steps - [Kubernetes](https://tryace.dev/docs/integrations/kubernetes): read pods, events and logs from the same cluster. - [AWS](https://tryace.dev/docs/integrations/aws): add CloudWatch, CloudTrail and infrastructure metadata. - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): how alerts become incidents. --- Source: https://tryace.dev/docs/integrations/grafana # Grafana > Connect Grafana so Grafana alerts open incidents in Ace, and Ace investigates them with your alert rules, dashboards and datasources. Source: https://tryace.dev/docs/integrations/grafana Ace reads Grafana alerts, rules, annotations and dashboards during investigations and queries your datasources, and Grafana-managed alerts open incidents. | | | | --- | --- | | Category | Dashboards | | Ace reads | Firing alerts, alert rules, annotations, dashboards, datasource queries and health | | Ace can change | Annotations when an incident opens and resolves, and silences on the paging alert rule while an approved fix rolls out | | Opens incidents | Yes, from Grafana alerts | | Setup | Settings form | Ace works with Grafana Cloud, Grafana Enterprise and self-hosted open-source Grafana. A self-hosted Grafana must use HTTPS and be reachable from the public internet, because Ace doesn't connect to private or internal addresses. ## What Ace reads - Active Grafana-managed alerts and alert rule definitions. - Annotations around the incident. - Dashboard search results, and a dashboard's panels and their queries. - One PromQL, LogQL, TraceQL or SQL query at a time against a datasource, over the incident window. - Datasource health. After a fix, Ace checks whether the incident's alert is still active. It also discovers your datasources and the backend each one reads, for the System Map. ## What Ace can change | Action | What it does | Approval | Undo | | --- | --- | --- | --- | | Annotation | When an incident opens or resolves, Ace adds an annotation such as "Incident opened: ``", tagged `ace`. It goes on the alert's dashboard when the alert links to this Grafana. Otherwise it's org-wide, with the workload's tag | Runs on its own | Yes. Undo from the incident deletes the annotation | | Silence | When an approved fix starts rolling out, Ace silences each alert rule that paged, for a bounded window. The silence matches only that rule, shows "Ace" as its creator and ends on its own | Runs on its own | Yes. Undo from the incident expires the silence | Ace makes these updates only after an admin turns on **Allow Ace to update this tool during incidents** for the connection. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). On Grafana Cloud and Enterprise, Ace reads the service account's permissions and shows which updates it can make. Open-source Grafana may not report them, so there a missing permission shows up only when Grafana refuses an update. ## What opens an incident Grafana alerts reach Ace through a webhook contact point as they fire. Ace doesn't poll Grafana, so nothing arrives until the webhook is set up. - A **firing** alert opens an incident or joins the one already open on the workload, and a **resolved** alert resolves it. - Severity comes from the `severity` label, which can be `critical` or `page`, `high` or `error`, `warning` or `medium`, and `info` or `low`. An alert without the label is high. - The route Ace creates skips alerts labeled `ace=ignore` and alerts with severity `info` or `none`. If you route alerts to Ace yourself, your notification policy decides what reaches Ace. An alert opens an incident only when Grafana is routed to a workload. To route it, open the workload's settings, turn on **Integration alerts** under **Investigation triggers** and select Grafana under **Alert sources**. ## Before you start - The **admin** role in Ace. See [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions). - The org **Admin** role in Grafana, to create a service account. - On open-source Grafana, the service account needs the Admin role to read alert rules, because Viewer and Editor can't. ## Credentials and permissions | Credential | Where to create it | Permissions | Why Ace needs it | | --- | --- | --- | --- | | Service account token (required) | **Administration** > **Users and access** > **Service accounts** | Viewer role and `alert.provisioning:read` | Reads alerts, dashboards, annotations, datasources and alert rules | | Same token (only for alerts) | | `alert.provisioning:write` | Adds Ace's contact point and notification route | | Same token (only for actions) | | `annotations:create` and `annotations:delete`, plus `alert.silences:create` and `alert.silences:write` (the `fixed:alerting.instances:writer` role, or Editor) | Adds and deletes annotations, and creates and expires silences | Ace uses one service account token. Grafana has no read-only provisioning role, so on Cloud and Enterprise, `fixed:alerting.provisioning:writer` grants `alert.provisioning:read` along with the write access that alert wiring uses. Don't use the secrets reader role, which exports decrypted secrets. ## Set up ### Create the service account token 1. In Grafana, open **Administration** > **Users and access** > **Service accounts** > **Add service account** (`https:///org/serviceaccounts/create`). 2. Set **Display name** to Ace and **Role** to Viewer, then choose **Create**. 3. On Grafana Cloud or Enterprise, add the fixed role `fixed:alerting.provisioning:writer`, and `fixed:annotations:writer` for incident annotations. On open-source Grafana, give the account the Admin role instead. 4. For silences, add `fixed:alerting.instances:writer`, or use the Editor role. 5. Choose **Add service account token** and **Set expiration date**, then **Generate token** and copy the token. ### Connect Grafana in Ace 1. In Ace, go to **Settings** > **Data sources** > **Grafana**. 2. Fill in the form: - **Grafana URL**: your Grafana's HTTPS URL, for example `https://example.grafana.net`. - **Service account token**: the token you copied. 3. Choose **Connect Grafana**. If any validation check fails, Ace doesn't save the token. ### Wire alerts If the token has `alert.provisioning:write`, Ace creates a webhook contact point and a catch-all notification route, both named `ace-`. The route continues to your other routes, so your existing routing is unchanged, and it also covers alert rules you add later. Both stay editable in Grafana, and the **Alerts** section shows what Ace set up. Ace checks it again every hour and removes it when you disconnect. To set it up yourself, choose **Show manual setup** in the **Alerts** section: 1. Copy the webhook URL Ace shows. 2. Choose **Generate token** and copy the token. Ace shows it once. 3. In Grafana, add a contact point with the **Webhook** integration and paste the URL. 4. Under **Optional Webhook settings**, leave **Authentication Header Scheme** at Bearer and paste the token into **Authentication Header Credentials**. 5. In a notification policy, route the alerts that should open incidents to this contact point. ### Allow updates (optional) On the connected Grafana page, turn on **Allow Ace to update this tool during incidents**. ## Confirm it's working - The connection shows **Connected** after Ace reads your dashboards and active alerts. - The **Alerts** section shows **Waiting for first event** until the first delivery Ace accepts, then **Verified**. ## Disconnect 1. In Ace, go to **Settings** > **Data sources** > **Grafana** and choose **Disconnect**. Ace removes its notification route and contact point, and if it can't, it tells you so you can remove `ace-` in Grafana yourself. 2. In Grafana, delete the service account or its token. ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): how observability updates and undo work. - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): how alerts become incidents. - [Integrations](https://tryace.dev/docs/integrations): connect more data sources. --- Source: https://tryace.dev/docs/integrations/new-relic # New Relic > Connect New Relic so alert issues open incidents in Ace, and Ace investigates them with your APM, OpenTelemetry, trace, error and NRQL data. Source: https://tryace.dev/docs/integrations/new-relic Ace reads golden metrics, traces, dependencies, Errors Inbox and NRQL from New Relic during investigations. Alert issues open incidents, and change-tracking deployments join the workload's change timeline. | | | | --- | --- | | Category | APM | | Ace reads | Alert issues, golden metrics, distributed traces, related entities, Errors Inbox, change-tracking deployments, NRQL results | | Ace can change | Muting rules for the condition that paged, while an approved fix rolls out | | Opens incidents | Yes, from alert issues | | Setup | Settings form | ## What Ace reads Ace reads through the NerdGraph API, and only in the accounts you list: - Alert issues around the incident, with title, priority, state, condition, policy, entities and the incidents inside each issue. - An entity's golden metrics, compared with the window before the incident. - Related entities: callers, downstream services, databases and hosts. - Distributed traces, one at a time. - Errors Inbox error groups, when your key can read them. New Relic's API doesn't return APM stack traces. - Change-tracking deployments before and during the incident. - NRQL, one query at a time over the incident window, including comparisons with an earlier period. After a fix, Ace checks the entity's error rate and latency. For the System Map, it discovers the APM applications and OpenTelemetry services in your accounts and maps the calls between them, along with the databases, external hosts and hosts they connect to. ## What Ace can change | Action | What it does | Approval | Undo | | --- | --- | --- | --- | | Muting rule | When an approved fix starts rolling out, Ace adds a muting rule named "Ace: fix rolling out" for each condition that paged, for a bounded window. The rule covers only that condition, never the whole entity, and ends on its own | Runs on its own | Yes. Undo from the incident removes the rule | This is the only change Ace makes in New Relic, and it makes it only after an admin turns on **Allow Ace to update this tool during incidents** for the connection. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). To mute a condition, Ace needs its `conditionFamilyId` and the entity GUIDs in the alert payload. The workflow Ace creates sends them, but Ace can't mute alerts from a manual workflow that leaves them out. ## What opens an incident Alert issues reach Ace through a workflow webhook as they fire. Ace also polls New Relic every 2 minutes, so a missed delivery still opens an incident. - An **activated** or **created** issue opens an incident or joins the one already open on the workload, and a **closed** issue resolves it. - Ace ignores acknowledgements and priority changes. - A **fully muted** issue never opens an incident, though a partially muted one does. A muted close doesn't count as recovery. - Ace ignores issues from accounts you didn't list. - To keep an issue out of Ace, put `ace:ignore` in its title, condition name or policy name. Severity comes from the issue's priority of Critical, High, Medium or Low. Change-tracking deployments go to the workload's change timeline and never open an incident. An issue opens an incident only when New Relic is routed to a workload. To route it, open the workload's settings, turn on **Integration alerts** under **Investigation triggers** and select New Relic under **Alert sources**. ## Before you start - The **admin** role in Ace. See [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions). - Permission in New Relic to create a user and grant it roles on each account. - For alert wiring and muting, a custom role for Ace's user, which needs New Relic Pro or Enterprise. ## Credentials and permissions Ace uses one user key. A user key has no scopes and acts as its user, so create a dedicated user for Ace with only the roles below. | Credential | Where to create it | Permissions | Why Ace needs it | | --- | --- | --- | --- | | User key (required) | **API keys** in New Relic, signed in as Ace's user | Read access to each account, and **Applied intelligence** > **Issues**: view | Reads telemetry and alert issues | | Same key (only for alerts) | | **Applied intelligence** > **Destinations**: modify, **Channels**: modify and delete, **Workflows**: modify and delete | Adds and maintains Ace's webhook destination, channel and workflow | | Same key (only for actions) | | **Alerts** > **Lifecycle overrides** (muting rules): modify and delete | Creates and removes muting rules | ## Set up ### Create the user key 1. In New Relic, create a user for Ace and grant it the standard **Read only** role on each account Ace should read, in **Administration** > **Access management** > **Access grants**. 2. For alert wiring and muting, add a custom role in **Roles** > **Add a role** with **Applied intelligence** > **Destinations**, **Channels** and **Workflows**, and **Alerts** > **Lifecycle overrides**, then grant it to Ace's user. 3. Signed in as Ace's user, open **API keys** (`https://one.newrelic.com/api-keys`, or `https://one.eu.newrelic.com/api-keys` in the EU) and choose **Create a key**. Set **Key type** to User, name it Ace and save it. 4. Copy the key from the row's menu. ### Connect New Relic in Ace 1. In Ace, go to **Settings** > **Data sources** > **New Relic**. 2. Fill in the form: - **User key**: the key you copied. It starts with `NRAK-`. - **Data center**: **US (api.newrelic.com)** or **EU (api.eu.newrelic.com)**. - **Account IDs**: the accounts Ace may read, alert on and write to, comma-separated. Each ID is in the account picker at the top of New Relic, or under **Administration** > **Access management** > **Accounts**. 3. Choose **Connect New Relic**. If any validation check fails, Ace doesn't save the key. ### Wire alerts If the key can manage destinations and workflows, Ace sets up a webhook destination, channel and workflow named `ace-` in each account, for up to 20 accounts. The workflow covers every issue. The **Alerts** section shows what Ace set up, and Ace checks it again every hour and removes it when you disconnect. To set it up yourself, choose **Show manual setup** in the **Alerts** section: 1. Copy the webhook URL Ace shows. 2. Choose **Generate token** and copy the token. Ace shows it once. 3. In New Relic, add a Webhook destination with the URL and Bearer token authorization. 4. Add a workflow that notifies the destination when issues are activated and closed. 5. Give the workflow a JSON payload with these fields: - `issueId`, `title`, `state`, `priority`, `issuePageUrl`, `triggerEvent` and `mutingState`. - `accountId`, set to the `nrAccountId` variable. - `conditionName`, `conditionFamilyId` and `policyName` from accumulations. - `entityNames`, `entityGuids` and `entityTypes` from `entitiesData`. ### Allow muting (optional) On the connected New Relic page, turn on **Allow Ace to update this tool during incidents**. ## Confirm it's working - The connection shows **Connected** after Ace checks that the key is a user key and sees each account in the data center you chose, runs a test NRQL query and reads issues. - The **Alerts** section shows **Waiting for first event** until the first delivery Ace accepts, then **Verified**. ## Disconnect 1. In Ace, go to **Settings** > **Data sources** > **New Relic** and choose **Disconnect**. Ace removes what it set up in each account, and if it can't, it tells you so you can remove `ace-` in New Relic yourself. 2. In New Relic, delete Ace's user key, or the Ace user. ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): how observability updates and undo work. - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): how alerts become incidents. - [Integrations](https://tryace.dev/docs/integrations): connect more data sources. --- Source: https://tryace.dev/docs/integrations/honeycomb # Honeycomb > Connect Honeycomb so triggers and SLO burn alerts open incidents in Ace, and Ace investigates them by querying your traces and events. Source: https://tryace.dev/docs/integrations/honeycomb Ace queries each service's traces and events in Honeycomb against a baseline and reads SLOs, triggers and markers. Triggers and SLO burn alerts open incidents, and deploy markers join the workload's change timeline. | | | | --- | --- | | Category | APM | | Ace reads | Columns, query results, traces, SLOs, triggers, markers | | Ace can change | Markers on the environment when an incident opens and resolves | | Opens incidents | Yes, from triggers and SLO burn alerts | | Setup | Settings form | ## What Ace reads Ace reads only the datasets you list, plus triggers, SLOs and markers that apply to the whole environment: - A dataset's columns, before it queries the dataset. - Queries over the incident window: counts, averages, percentiles, heatmaps, distinct counts and rates. - Comparisons of failing events against the rest, to find the route, version, host or customer they share. - Traces, one at a time. - SLOs and their burn, triggers and whether they fired, and markers around the incident. Queries, comparisons and traces need **Run Queries**, which Honeycomb offers on Enterprise. Without it, Ace still reads triggers, SLOs and markers. After a fix, Ace queries the incident's dataset to confirm the errors stopped. For the System Map, it discovers your datasets and, on Enterprise, the service map's dependencies and the routes each service serves. ## What Ace can change | Action | What it does | Approval | Undo | | --- | --- | --- | --- | | Marker | When an incident opens or resolves, Ace adds one environment-wide marker so your charts show it, for example "Incident opened: `` (workload ``)" | Runs on its own | Yes. Undo from the incident deletes the marker | Ace adds markers only after an admin turns on **Allow Ace to update this tool during incidents** for the connection, and only when the key has **Manage Markers**. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). ## What opens an incident Triggers and burn alerts reach Ace through a webhook recipient as they fire. While Honeycomb is routed to a workload, Ace also polls every 2 minutes for fired triggers and burn alerts in the environment and your datasets, so a missed delivery still opens an incident. - A **TRIGGERED** notification opens an incident or joins the one already open on the workload, and **OK** resolves it. - An SLO resolves once none of its burn alerts is still triggered. - Ace ignores test notifications. - Ace ignores alerts that name only datasets you didn't list, but accepts alerts for the whole environment. - To keep a trigger, SLO or burn alert out of Ace, such as one for an audit or business query, put `ace:ignore` in its name or description. Severity comes from a `severity`, `priority` or `sev` tag, and without one, Ace sets it from the alert itself. Deploy markers on the environment and your datasets go to the workload's change timeline and never open an incident. An alert opens an incident only when Honeycomb is routed to a workload. To route it, open the workload's settings, turn on **Integration alerts** under **Investigation triggers** and select Honeycomb under **Alert sources**. ## Before you start - The **admin** role in Ace. See [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions). - The **Team Owner** role in Honeycomb, to create a configuration key. - Honeycomb Enterprise for queries, comparisons, traces and service map discovery. ## Credentials and permissions Ace uses one configuration key for one environment. Ingest keys don't work. | Credential | Where to create it | Permissions | Why Ace needs it | | --- | --- | --- | --- | | Configuration key (required) | **Manage Environments** > the environment > **API Keys** > **Configuration** | **Manage Queries and Columns**, **Run Queries** (Enterprise), **Manage SLOs**, **Manage Triggers**, **Manage Markers** | Reads columns, runs queries, and reads SLOs, triggers and markers | | Same key (only for alerts) | | **Manage Recipients** | Adds Ace's webhook recipient to triggers and burn alerts. Recipients are team-wide | | Same key (only for actions) | | **Manage Markers** | Adds and deletes incident markers | ## Set up ### Create the configuration key 1. As a Team Owner, open the Environments selector > **Manage Environments** > the environment > **API Keys** > **Configuration** > **Create Configuration API Key**. 2. Name it Ace and select **Manage Queries and Columns**, **Run Queries**, **Manage SLOs**, **Manage Triggers** and **Manage Markers**. Add **Manage Recipients** if Ace should wire alerts, knowing that this permission changes recipients for the whole team. 3. Create the key and copy it. ### Connect Honeycomb in Ace 1. In Ace, go to **Settings** > **Data sources** > **Honeycomb**. 2. Fill in the form: - **Configuration key**: the key you copied. - **Region**: **US (api.honeycomb.io)** or **EU (api.eu1.honeycomb.io)**. - **Datasets**: the datasets (services) Ace may query, alert on and mark, comma-separated. 3. Choose **Connect Honeycomb**. If any validation check fails, Ace doesn't save the key. ### Wire alerts If the key has **Manage Recipients**, Ace adds a webhook recipient to every trigger and burn alert of the environment and your datasets, skipping those marked `ace:ignore`. Ace checks again every hour to pick up new triggers, and the **Alerts** section shows what Ace set up. To set it up yourself, choose **Show manual setup** in the **Alerts** section: 1. Copy the webhook URL Ace shows. 2. Choose **Generate token** and copy the token. Ace shows it once. 3. In Honeycomb, open **Team Settings** > **Integrations**. Under **Trigger, SLOs, and Anomaly Detection Recipients**, choose **Add Integration**, set **Provider** to Webhook, enter the URL and the token as its **Shared Secret**, and choose **Add**. 4. Add the recipient to the triggers and burn alerts that should open incidents. ### Allow markers (optional) On the connected Honeycomb page, turn on **Allow Ace to update this tool during incidents**. ## Confirm it's working - The connection shows **Connected** after Ace checks that the key is a configuration key with each permission it needs and that it can find each dataset you listed. - The **Alerts** section shows **Waiting for first event** until the first delivery Ace accepts, then **Verified**. ## Disconnect 1. In Ace, go to **Settings** > **Data sources** > **Honeycomb** and choose **Disconnect**. Ace removes its recipient from your triggers and burn alerts and deletes it, except where a burn alert notifies only Ace, in which case Ace leaves the recipient in place. When Ace can't remove something, it tells you so you can remove `ace-` in Honeycomb yourself. 2. In Honeycomb, delete the configuration key. ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): how observability updates and undo work. - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): how alerts become incidents. - [Integrations](https://tryace.dev/docs/integrations): connect more data sources. --- Source: https://tryace.dev/docs/integrations/elasticsearch # Elasticsearch > Connect Elasticsearch so Ace can search your logs, metrics and traces during investigations. Source: https://tryace.dev/docs/integrations/elasticsearch Ace runs bounded, read-only searches against the indices your API key can read, looking for the errors and log lines around an incident. | | | | --- | --- | | Category | Logs, metrics and traces via Elasticsearch | | Ace reads | Log, metric and trace documents, index and field metadata | | Ace can change | Nothing (read-only) | | Opens incidents | No | | Setup | Settings form | ## What Ace reads When you connect, Ace discovers your indices and data streams with their timestamp fields and most common fields, and samples values for fields such as service, namespace, environment, host and severity. It also checks whether your cluster supports search, ES|QL and field capabilities. During an investigation, Ace runs searches over the incident window: - Recent log lines at every level, and recent error logs, with message, severity, service, environment, namespace, host and trace ID. - Error rate over time. - The services with the most errors. - An ES|QL error summary, when your cluster supports ES|QL. - Recent metric and trace documents. Ace uses Elasticsearch in investigations only after validation passes and discovery completes. ## What Ace can change Ace changes nothing in Elasticsearch, so the API key needs only read privileges. ## What opens an incident Elasticsearch doesn't open incidents, and Ace uses it only as evidence. To open incidents from alerts, connect an alert source such as [Datadog](https://tryace.dev/docs/integrations/datadog) or [Prometheus](https://tryace.dev/docs/integrations/prometheus). ## Before you start - The **admin** role in Ace. - An Elasticsearch endpoint reachable over HTTPS. - Permission in Kibana to create an API key. Ace uses one Elasticsearch connection per workspace. ### Private networks Ace connects from its cloud to one HTTPS endpoint URL, with no agent for your network and no proxy or private-link option. Elastic Cloud endpoints work as they are. A self-hosted cluster, or one in a VPC, needs an HTTPS endpoint reachable from outside your network, and Ace doesn't publish addresses to allowlist. If your cluster is reachable only inside a private network, contact our team before you connect. ## Credentials and permissions | Credential | Where to create it | Privileges | Why Ace needs it | | --- | --- | --- | --- | | API key | Kibana: **Stack Management** > **API keys** > **Create API key** | Index privileges `read` and `view_index_metadata` on the indices Ace should search | `read` runs searches. `view_index_metadata` discovers indices, data streams and fields | When you create the key, turn on **Control security privileges** and grant only these privileges, on the indices Ace should search. ## Set up 1. Find your Elasticsearch endpoint: - In Elastic Cloud, open your deployment or project > **Manage** and choose **Copy endpoint** next to Elasticsearch. - In Kibana, open **Help** > **Connection details** > **Endpoints**. Use the Elasticsearch endpoint, not the Cloud ID. 2. Create the API key in Kibana, as described above. 3. In Ace, go to **Settings** > **Data sources** > **Elasticsearch**. 4. Enter the **Endpoint URL**, which must start with `https://`, and the **API key**. 5. Choose **Connect Elasticsearch**. To change the endpoint or key, choose **Update credentials**, then **Save and revalidate**. Ace replaces the stored configuration, validates it again and refreshes discovery. ## Confirm it's working - The panel shows **Validating integration…** while Ace checks that the key authenticates and the cluster is healthy, and then **Connected**. - After discovery, the panel lists the indexed data Ace found. If it shows **No data indexed yet**, wait for discovery to finish or check the key's index privileges. - If a check fails, fix the cause and choose **Re-validate**. ## Disconnect In Ace, choose **Disconnect** in **Settings** > **Data sources** > **Elasticsearch**, then invalidate Ace's key in Kibana under **Stack Management** > **API keys**. ## Next steps - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): how evidence feeds an investigation. - [Security and data](https://tryace.dev/docs/security/security-and-data): what Ace stores. - [Integrations](https://tryace.dev/docs/integrations): connect an alert source. --- Source: https://tryace.dev/docs/integrations/axiom # Axiom > Connect Axiom so monitor alerts open incidents in Ace, and Ace investigates them by querying your logs and events with APL. Source: https://tryace.dev/docs/integrations/axiom Ace queries logs and events in the Axiom datasets you choose, with bounded APL over the incident window, and monitor alerts open incidents. | | | | --- | --- | | Category | Logs | | Ace reads | Event volume, text search and APL results in your datasets, and monitors and their notifications | | Ace can change | Annotations on your datasets when an incident opens and resolves | | Opens incidents | Yes, from monitor alerts | | Setup | Settings form | ## What Ace reads Ace reads only the datasets you list and never reads or lists other datasets, including for the System Map. Each query runs on one dataset over the incident window and returns a limited number of rows, and Ace refuses a query that reaches another dataset, for example through a union, join or lookup. - Event counts over time in one dataset, optionally split by a field. - Case-insensitive text search, newest first. - One APL pipeline over one dataset. - Monitors on your datasets, with their query, threshold and the notifications they sent in the window. ## What Ace can change | Action | What it does | Approval | Undo | | --- | --- | --- | --- | | Annotation | When an incident opens or resolves, Ace adds one annotation to the connection's datasets so your charts show it, for example "Incident opened: ``" | Runs on its own | Yes. Undo from the incident deletes the annotation | Ace adds annotations only after an admin turns on **Allow Ace to update this tool during incidents** for the connection. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). Ace can't check annotation permissions in advance, so Settings shows **Shows when it runs: needs Annotations: create, delete**, and a missing permission shows up only when Axiom refuses the update. ## What opens an incident Monitor alerts reach Ace through a custom webhook notifier as they fire. Ace doesn't poll Axiom, so nothing arrives until the notifier is set up. - A notification with action **Open** opens a high-severity incident or joins the one already open on the workload, and **Closed** resolves it. Ace ignores other notifications. - When Ace adds its notifier, it skips monitors with `ace:ignore` in their name or description. An alert opens an incident only when Axiom is routed to a workload. To route it, open the workload's settings, turn on **Integration alerts** under **Investigation triggers** and select Axiom under **Alert sources**. ## Before you start - The **admin** role in Ace. See [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions). - Permission in Axiom to create API tokens. You can't change a token's permissions later, so if you want alert wiring or annotations, add those permissions when you create the token. ## Credentials and permissions Ace uses one API token, with the permissions below. | Credential | Where to create it | Permissions | Why Ace needs it | | --- | --- | --- | --- | | API token (required) | **Settings** > **API tokens** | Query on each dataset you list, and **Datasets**: read | Queries your datasets and lists them | | Same token (only for alerts) | | **Monitors**: read, update, and **Notifiers**: create, read, update, delete | Adds Ace's notifier to your monitors | | Same token (only for actions) | | **Annotations**: create, delete | Adds and deletes incident annotations | ## Set up ### Create the API token 1. In Axiom, open **Settings** > **API tokens** > **New API token**, name it Ace and set an expiration date. 2. Under **Token permissions**, choose **Advanced**, then **Custom**, and add Query on each dataset Ace should read. Under **Org level permissions**, add **Datasets**: Read. 3. If Ace should wire monitor alerts, add **Monitors** (read, update) and **Notifiers** (create, read, update, delete). For incident annotations, add **Annotations** (create, delete). 4. Optionally set query cost limits, then create the token and copy it. ### Connect Axiom in Ace 1. In Ace, go to **Settings** > **Data sources** > **Axiom**. 2. Fill in the form: - **API token**: the token you copied. - **Organization ID** (optional): required for personal tokens. Find it under **Settings** > **General** > **ID**, or after `app.axiom.co/` in the app URL. - **Datasets**: the datasets Ace may read, comma-separated. List at least one. - **API URL** (optional): only if you don't use `https://api.axiom.co`. It must use HTTPS and be reachable from the public internet, because Ace doesn't connect to private or internal network addresses. 3. Choose **Connect Axiom**. Ace saves the token only if it can see each dataset. ### Wire alerts If the token can manage monitors and notifiers, Ace creates a custom webhook notifier named `ace-` and adds it to every monitor except those marked `ace:ignore`. Ace checks again every hour to pick up new monitors, and removes the notifier when you disconnect. > **Adding a notifier resets a monitor's alert state** > > Axiom replaces a monitor when its notifiers change, so every monitor that gets Ace's notifier starts its alert state over. To set it up yourself, choose **Show manual setup** in the **Alerts** section: 1. Copy the webhook URL Ace shows. 2. Choose **Generate token** and copy the token. Ace shows it once. 3. In Axiom, open **Monitors** > **Manage notifiers** > **New notifier**, choose **Custom webhook**, enter the URL as the **Webhook URL**, add an `Authorization` header of `Bearer ` and choose **Create**. 4. Add the notifier to the monitors that should open incidents. ### Allow annotations (optional) On the connected Axiom page, turn on **Allow Ace to update this tool during incidents**. ## Confirm it's working - The connection shows **Connected** and the number of datasets the token sees. - The **Alerts** section shows **Waiting for first event** until the first delivery Ace accepts, then **Verified**. ## Disconnect 1. In Ace, go to **Settings** > **Data sources** > **Axiom** and choose **Disconnect**. Ace removes its notifier from your monitors and deletes it, and if it can't, it tells you so you can remove `ace-` in Axiom yourself. 2. In Axiom, delete the API token. ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): how observability updates and undo work. - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): how alerts become incidents. - [Integrations](https://tryace.dev/docs/integrations): connect more data sources. --- Source: https://tryace.dev/docs/integrations/logfire # Logfire > Connect Pydantic Logfire so Ace can query your application logs, traces, exceptions and metrics during investigations. Source: https://tryace.dev/docs/integrations/logfire Ace runs bounded, read-only SQL queries against Logfire with a project read token, so it sees your application logs, traces and exceptions without being able to change anything. | | | | --- | --- | | Category | Application logs, traces and exceptions | | Ace reads | Log and span records, exceptions and metric point counts | | Ace can change | Nothing (read-only) | | Opens incidents | No | | Setup | Settings form | ## What Ace reads During an investigation, Ace queries the incident window for: - Record counts by service, environment and level. - The latest error and exception records. - Metric point counts. If you set an **Environment allowlist**, Ace investigates only those environments. ## What Ace can change Ace changes nothing in Logfire, because a Logfire read token can only read. ## What opens an incident Logfire doesn't open incidents, and Ace uses it only as evidence. ## Before you start - The **admin** role in Ace. - A read token for your Logfire project. Logfire cloud in the US or EU works as it is. For a self-hosted or dedicated Logfire, you enter a custom HTTPS endpoint, which must be reachable over HTTPS from outside your network because Ace calls it from its cloud. ## Credentials and permissions | Credential | Where to create it | Permissions | Why Ace needs it | | --- | --- | --- | --- | | Read token | Logfire project: **Settings** > **Read tokens** > **Create read token** | Read access to the project | Runs bounded SQL queries over your records | Logfire shows the token only once, so copy it before you close the dialog. ## Set up 1. Create the read token in your Logfire project. 2. In Ace, go to **Settings** > **Data sources** > **Logfire**. 3. Fill in the form: - **Region**: **US (https://logfire-us.pydantic.dev)** or **EU (https://logfire-eu.pydantic.dev)**, to match where your project stores data. For a self-hosted or dedicated deployment, choose **Custom** and enter the **Custom endpoint URL**, which must use HTTPS. - **Read token**: starts with `pylf_v1_`. - **Environment allowlist** (optional): environment names separated by commas, semicolons or new lines. Leave it blank to investigate every environment the token can see. 4. Choose **Connect Logfire**. To change the configuration, choose **Update credentials**, then **Save and revalidate**. Ace replaces the stored configuration, validates it again and refreshes discovery. ## Confirm it's working - Ace checks that the token is valid and can query records, and then the panel shows **Connected** and the discovery panel shows what Ace found. - If a check fails, fix the cause and choose **Re-validate**. ## Disconnect In Ace, choose **Disconnect** in **Settings** > **Data sources** > **Logfire**, then revoke Ace's token in your Logfire project under **Settings** > **Read tokens**. ## Next steps - [LangSmith](https://tryace.dev/docs/integrations/langsmith): add AI workflow traces. - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): how evidence feeds an investigation. - [Security and data](https://tryace.dev/docs/security/security-and-data): what Ace stores. --- Source: https://tryace.dev/docs/integrations/sentry # Sentry > Connect Sentry so issue and metric alerts open incidents in Ace, and Ace reads errors, releases, spans, logs and replays while it investigates. Source: https://tryace.dev/docs/integrations/sentry Sentry alerts open incidents in Ace, and Ace reads the errors behind them while it investigates. Repeated events of one Sentry issue land in the same incident, so a noisy error doesn't open a new incident each time it fires. | | | | --- | --- | | Category | Errors | | Ace reads | Unresolved issues, latest events, releases, Explore queries over errors, spans and logs, session replays, projects | | Ace can change | Issue status and assignee, only after an admin allows it | | Opens incidents | Yes, for workloads that route Sentry alerts | | Setup | Settings form, plus a webhook on a Sentry internal integration | ## What Ace reads Ace reads Sentry while it investigates a workload that lists Sentry under **Investigation sources** or routes its alerts. Every read covers the incident window and stays inside your project allowlist, if you set one. - Unresolved issues in the window, most frequent first. - The latest event of an issue. Ace gets exception types and messages, in-app stack frames, the release, and only these tags: `environment`, `release`, `transaction`, `url`, `server_name` and `runtime`. - Recent releases, with dates and the number of new issues each one introduced. - Explore table and aggregate queries over errors, spans and logs. - Session replays with errors. Replays carry user identity, so Ace gets only the pages visited (without query strings), environment, browser, OS and error IDs. - New events of the incident's issue after a fix, to confirm the errors stopped. Ace also adds your Sentry projects to the System Map. ## What Ace can change These writes change what your team sees in Sentry and never touch code or infrastructure. They need extra token permissions and run only after an admin turns on **Allow Ace to update this tool during incidents** for the connection, and from then on they run without an approval step. | Action | When it runs | Undo | | --- | --- | --- | | Resolve issue | Ace resolves the incident's Sentry issues once it verifies the incident recovered. | Yes. Undo reopens the issue. | | Assign issue | When an approved fix starts, Ace assigns the incident's Sentry issues to the approver, matched by email. | Yes | Ace skips issues outside your project allowlist, and it leaves an issue alone if someone changed its status in the meantime, so it doesn't overwrite your team's triage. You undo each change from the incident. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals) for how actions work in general. ## What opens an incident Sentry alerts open incidents only on workloads that route this connection (**Integration alerts** on, Sentry selected). Ace drops alerts that no workload routes. | Sentry sends | What Ace does | | --- | --- | | An alert rule action (issue alert) | Opens or joins an incident for that Sentry issue. Fatal events are critical, errors high, warnings medium and anything else low. | | Issue `created` or `unresolved` | Opens or joins an incident, but only for issues at level error or fatal. User feedback issues never open incidents. | | Issue `resolved`, `ignored` or `archived` | Marks that issue resolved in Ace. An incident resolves once nothing in it is still firing. | | Metric alert | Opens or joins an incident. A `critical` trigger is critical severity and anything else is high. A resolved metric alert resolves it. | Ace ignores every other delivery, such as issue assignments. Ace also polls Sentry for new unresolved issues at level error or fatal, so a missed webhook doesn't hide a new failure. Polling never opens incidents for warnings, info or user feedback. ## Before you start - The **admin** role in Ace. - The **Manager** or **Owner** role in your Sentry organization, to create an internal integration. Organization auth tokens can't read events, so use an internal integration. ## Credentials and permissions Both credentials come from one Sentry internal integration. Leave every permission not listed here at No Access. | Credential | Where to create it | Permissions | Why Ace needs it | | --- | --- | --- | --- | | Auth token (needed) | **Settings** > **Custom Integrations** > **Create New Integration** > **Internal Integration**, under **Tokens** | Project: Read, Release: Admin, Issue & Event: Read, Organization: Read (scopes `org:read`, `project:read`, `event:read`, `project:releases`) | Read issues, events, releases, Explore data and replays. Release: Admin is the only level that grants `project:releases`. | | Auth token (only for issue updates) | Same integration | Issue & Event: Read & Write, Member: Read (`event:write`, `member:read`) | Resolve and reopen issues, and look up the approver to assign issues. | | Client secret (only for alerts) | Same integration, under **Credentials** | None | Verifies that webhook deliveries come from Sentry. | ## Set up ### 1. Create the internal integration in Sentry 1. Go to **Settings** > **Custom Integrations** > **Create New Integration** > **Internal Integration**. 2. Name it Ace. Under **Permissions**, set Project: Read, Release: Admin, Issue & Event: Read and Organization: Read. For issue updates, set Issue & Event: Read & Write and Member: Read instead. 3. Select **Save Changes**, then copy the token under **Tokens** and the Client Secret under **Credentials** right away. Sentry shows the client secret only briefly after creation, and **Rotate client secret** issues a new one later. ### 2. Connect Sentry in Ace 1. Open **Settings** > **Data sources** and choose **Sentry**. 2. Fill in the form: | Field | What to enter | | --- | --- | | **Auth token** | The token from the internal integration. | | **Organization slug** | Your Sentry organization slug. | | **Sentry URL** | Optional. `https://de.sentry.io` for organizations hosted in the EU, or your self-hosted URL. Leave blank for `https://sentry.io`. | | **Projects** | Optional. A comma-separated allowlist of project slugs. | | **Client secret** | Optional. The client secret from the same integration. Ace needs it to verify alert webhooks. | 3. Select **Connect Sentry**. Ace checks the token before it saves. A self-hosted Sentry URL must use HTTPS and be reachable from the public internet, because Ace doesn't connect to private or internal network addresses. ### 3. Send alerts to Ace 1. In the **Alerts** section, copy the webhook URL. 2. In Sentry, open the internal integration and paste it as the **Webhook URL**. 3. Turn on **Alert Rule Action**, subscribe to **issue**, and save. 4. In **Monitors** > **Alerts**, open or create each alert that should open incidents. Under **Then**, add the Ace integration. ### 4. Route alerts to a workload Open the workload in Ace, turn on **Integration alerts**, and select Sentry as a source. ### 5. Optional: allow issue updates If the token has the write permissions, turn on **Allow Ace to update this tool during incidents** on the connection. ## Confirm it's working - The connection's checks read the organization and report how many projects the token sees. If the token can't see a project in your allowlist, the check names it. - The **Alerts** badge shows **Not set up**, **Waiting for first event**, **Verified** or **Last delivery failed**. Send a test alert from Sentry, and the badge turns **Verified** when the first signed delivery arrives. - A failed delivery shows the reason, such as a signature that doesn't match the saved client secret. - **Re-validate** reruns the checks. ## Disconnect 1. In Ace, open the Sentry connection under **Settings** > **Data sources** and select **Disconnect**. To stop issue updates and keep the connection, turn off **Allow Ace to update this tool during incidents** instead. 2. In Sentry, delete the Ace internal integration or its token. Ace creates nothing else in Sentry. ## Next steps - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works) - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals) - [All integrations](https://tryace.dev/docs/integrations) --- Source: https://tryace.dev/docs/integrations/langsmith # LangSmith > Connect LangSmith so Ace can read your AI workflow runs and traces during investigations. Source: https://tryace.dev/docs/integrations/langsmith Ace reads LangSmith to see what your AI workflows did around an incident, including which runs failed, how long they took and what their child spans did. | | | | --- | --- | | Category | AI workflow traces, latency, cost and feedback | | Ace reads | Projects, root runs, failed runs, child runs (spans) and project feedback stats | | Ace can change | Nothing (read-only) | | Opens incidents | No | | Setup | Settings form | ## What Ace reads When you connect, Ace lists the projects the API key can see, with their stats. During an investigation it reads: - Project runs in the incident window: root runs, failed root runs, runs that match the incident, and their child spans. - The child runs (spans) of one trace, in start order. **Workspace ID** limits Ace to one workspace, and **Project allowlist** limits it to specific projects. ## What Ace can change Nothing. ## What opens an incident Nothing. Ace uses LangSmith as evidence only. ## Before you start - The **admin** role in Ace. - A LangSmith API key that can list projects. - LangSmith cloud in the US or EU works as is. A dedicated or private base URL goes in as a custom endpoint, which must be reachable over HTTPS from Ace's cloud. ## Credentials and permissions | Credential | Where to create it | Permissions | Why Ace needs it | | --- | --- | --- | --- | | API key | LangSmith: **Settings** > **API Keys** > **Create API Key** | Read access to the projects Ace should see. A service key scoped to your workspace works best. On Enterprise, give it the **Workspace Viewer** role so it can only read. | List projects and read runs. | ## Set up 1. Create the API key in LangSmith. 2. In Ace, go to **Settings** > **Data sources** > **LangSmith**. 3. Fill in the form: - **Endpoint**: **US (https://api.smith.langchain.com)** or **EU (https://eu.api.smith.langchain.com)**, matching your workspace's region. Choose **Custom** only if LangSmith gave you a dedicated or private base URL, which must use HTTPS. - **API key**: starts with `lsv2_`. - **Workspace ID** (optional): the workspace Ace should use, if the key can access several. Find it in LangSmith under **Settings** > **General**. - **Project allowlist** (optional): one project per line. Leave it blank to let Ace use every project the key can read. 4. Choose **Connect LangSmith**. Ace validates the key first, and if a check fails it shows the error and doesn't connect. To change the configuration, choose **Update credentials**, then **Save and revalidate**. Ace replaces the stored configuration, revalidates and refreshes discovery. ## Confirm it's working The panel shows **Connected**, and the discovery panel lists your projects. Two validation errors come up most often: - "No LangSmith projects were visible to this API key. If this is an org-scoped key, add the workspace ID." - "No LangSmith projects matched the configured project allowlist." Check the names in **Project allowlist**. ## Disconnect In Ace, go to **Settings** > **Data sources** > **LangSmith** and choose **Disconnect**. In LangSmith, delete the key under **Settings** > **API Keys**. ## Next steps - [Logfire](https://tryace.dev/docs/integrations/logfire): add application logs and traces. - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): how evidence feeds an investigation. - [Security and data](https://tryace.dev/docs/security/security-and-data): what Ace stores. --- Source: https://tryace.dev/docs/integrations/litellm # LiteLLM > Connect your LiteLLM proxy so Ace reads model errors and spend during investigations, finds wasted model spend, and proposes fallback and budget changes for approval. Source: https://tryace.dev/docs/integrations/litellm Ace reads your LiteLLM proxy during an incident for per-model calls, failures, error classes, tokens, spend and latency. It also scans the proxy for duplicate and wasted model spend. | | | | --- | --- | | Category | LLM gateway | | Ace reads | Spend logs (metadata only), daily spend, model deployments | | Ace can change | Model fallbacks, and a virtual key's or team's budget and rate limits, each after approval | | Opens incidents | No (evidence only) | | Setup | Settings form | ## What Ace reads - Calls by model in the incident window: calls, failures, error classes, tokens, spend and average latency. - The latest failed calls, with error class and a short error message. - Daily spend across the proxy. From spend logs Ace keeps metadata only: model, spend, tokens, times, user, key hash and alias, cache key and hit, status, and error class and message. It never holds prompts or responses. Ace adds the model deployments behind the proxy to the System Map, with their keys masked by LiteLLM. ### Cost findings Ace reports two kinds of waste from your spend logs and shows how it calculated each one. - **Repeated identical LLM requests**: many calls that repeat the same request. Ace can't fix this in your application, so turn on response caching in LiteLLM or stop the calling service from retrying completed work. - **Spend on failed LLM calls**: calls the provider charged for that returned an error. ## What Ace can change Ace proposes these actions once an admin turns on **Allow Ace to run approved actions** for the connection. Each one needs approval by default, and earned autonomy can let the fallback action run under an Automatic policy on a workload and environment you confirm. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). | Action | What it does | Approval | Undo | | --- | --- | --- | --- | | LiteLLM model fallback | Sets or removes one model's fallbacks, so a failing model's requests go to other models. Ace refuses unknown models, a model as its own fallback and duplicates. It also refuses to apply if the fallbacks changed since approval. | Needs approval, can earn Automatic | Yes. Restores the fallbacks it replaced. | | LiteLLM budget and limits | Changes a virtual key's or team's max budget and its TPM and RPM limits. Ace names keys by their hashed token, never the secret key. LiteLLM records the change as made by Ace. | Always needs approval | Yes. Restores the values it replaced. | Both actions need the Admin role in LiteLLM. Fallback changes also need `STORE_MODEL_IN_DB=True` on the proxy and no router fallbacks in `config.yaml`, or the proxy refuses them. ## What opens an incident Nothing. LiteLLM sends Ace no alerts, and Ace doesn't poll it for failures. To use LiteLLM as evidence, turn it on under **Investigation sources** for each workload that calls it. ## Before you start - The **admin** role in Ace. See [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions). - **Proxy Admin** in LiteLLM, to create Ace's user and key. - A proxy on HTTPS that is reachable from the public internet, because Ace doesn't connect to private or internal network addresses. ## Credentials and permissions Ace uses one virtual key, owned by a LiteLLM user you create for Ace. Keep the key type at Full Access, because AI APIs keys can't read spend logs. | Credential | Where to create it | Role | Why Ace needs it | | --- | --- | --- | --- | | Admin key (needed) | **Virtual Keys** in the LiteLLM UI, owned by Ace's user | Ace's user has Global Proxy Role **Admin (View Only)** (`proxy_admin_viewer`) | Read spend logs, daily spend and model deployments. | | Same key (only for actions) | Same | Ace's user has Global Proxy Role **Admin** (`proxy_admin`) | Change fallbacks, budgets and rate limits after approval. | ## Set up ### 1. Create Ace's user and key 1. As a Proxy Admin, open **Internal Users** > **Invite User**. Enter an email for Ace and set **Global Proxy Role** to Admin (View Only), or Admin for approved changes. 2. In Ace, open **Settings** > **Data sources** and choose **LiteLLM**. Enter your **Proxy URL**, then select **Create key** next to **Admin key**. 3. LiteLLM opens **Virtual Keys** > **Create New Key** with Owned By set to Another User, Key Name set to Ace and Key Type set to Full Access. Select Ace's user. 4. Optionally set an expiry under **Key Lifecycle**. Select **Create Key** and copy the key. Proxies older than June 2026 open `/ui/?page=api-keys` without these presets, so set the same values by hand. ### 2. Connect LiteLLM in Ace 1. Fill in the form: - **Proxy URL**: your proxy's HTTPS URL, for example `https://litellm.example.com`. - **Admin key**: the key you copied. 2. Select **Connect LiteLLM**. Ace saves only if the proxy is ready and the key can read spend logs. ### 3. Optional: allow actions 1. On the connected LiteLLM page, turn on **Allow Ace to run approved actions**. 2. For automatic fallback changes, select a workload, enter its environment (default `production`), and select **Confirm**. ## Confirm it's working - The connection shows **Connected** once the readiness and spend-log checks pass. A key without the viewer role fails with "The key is missing the proxy_admin_viewer role." - LiteLLM is on under **Investigation sources** for each workload that calls it. ## Disconnect 1. In Ace, open **Settings** > **Data sources** > **LiteLLM** and select **Disconnect**. Ace set nothing up in LiteLLM, so there is nothing else to remove there. 2. In LiteLLM, delete Ace's virtual key or Ace's user. ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals) - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works) - [All integrations](https://tryace.dev/docs/integrations) --- Source: https://tryace.dev/docs/integrations/vercel # Vercel > Connect Vercel so your deploys join the change ledger next to incidents, and Ace can propose rollbacks, promotions, redeploys and firewall changes. Source: https://tryace.dev/docs/integrations/vercel Every Vercel deploy joins the change ledger, Ace's per-workload record of deploys and config changes. Ace uses it to tell whether an incident started right after a release, and it can roll back or promote a deployment after approval. | | | | --- | --- | | Category | Deploys | | Ace reads | Deployments, build events, projects, domains, regions, environment variables that aren't secret | | Ace can change | Rollback, promotion, redeploy, cancel, rolling releases, Attack Challenge Mode, firewall IP blocks, environment variables that aren't secret, only after an admin allows actions | | Opens incidents | Yes, from Vercel alerts. Deploys also join the change ledger. | | Setup | Settings form, plus an optional Vercel deployment webhook | ## What Ace reads While it investigates a workload with Vercel under **Investigation sources**, Ace reads: - Recent deployments around the incident, production by default and newest first, with commit SHA, commit message, branch and creator. - The latest build events of a deployment. For the System Map, Ace reads projects, function regions, the live production deployment, domains and environment variables. It reads only variables that aren't marked Sensitive or secret, to find the hosts and databases they point to. It never reads or writes Sensitive or secret variables, so keep sensitive values in those. A **Projects** allowlist limits all reads to those projects. ## What Ace can change Ace proposes Vercel actions only after an admin turns on **Allow Ace to run approved actions** for the connection. Each one needs approval by default, and earned autonomy can let some of them run under an Automatic policy for a workload and environment you confirm. Others always need a person. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). | Action | What it does | Approval | Undo | | --- | --- | --- | --- | | Instant rollback | Serves a previous ready production deployment again, without a rebuild. | Approval, or Automatic under your policy | Yes. Promotes the previous deployment again. | | Promotion | Makes a ready production deployment the one serving production, without a rebuild. | Approval, or Automatic under your policy | Yes | | Redeploy | Rebuilds a deployment in production, preview or a custom environment. | Approval, or Automatic under your policy | Yes | | Cancel deployment | Cancels a build that is still queued or building. | Approval, or Automatic under your policy | No | | Rolling release abort | Aborts a rolling release with an instant rollback. | Approval, or Automatic under your policy | No | | Rolling release stage approval | Approves the next stage of a rolling release. | Always a person | No | | Attack Challenge Mode | Turns it on for the project for a set time (15 to 1,440 minutes, 60 by default), or off. | Always a person | Yes | | Firewall IP block or unblock | Adds or removes a firewall IP rule. | Always a person | Yes | | Environment variable change | Sets one variable that isn't secret to a value a person enters, then redeploys production so it takes effect. | Always a person | Yes. Restores the previous value and redeploys. | Ace applies these safeguards: - Rollback and promotion switch only to a ready production deployment of the same project. Before a rollback, Ace checks the code it would remove from production (through the GitHub App when connected) and can block an unsafe rollback. If production changed after approval, Ace refuses to run. - After a rollback, Vercel stops assigning production domains to new builds on its own. When your team's next production build arrives, Ace offers to promote it on the original incident. - Ace changes only environment variables that apply to production, aren't tied to one Git branch and aren't secret, and it never reads or writes Sensitive, secret or system variables. A person always enters the new value, and Ace keeps the previous value for undo without showing either one in the plan. - Each rollback Ace makes carries a description in Vercel that says whether a person or your action policy approved it. Vercel tokens have no write scopes, so actions follow the team role of the token's owner (see [Credentials and permissions](#credentials-and-permissions)). ## What opens an incident Vercel alerts open an incident in Ace the moment they fire, on the workload that routes Vercel. Ace also polls Vercel, so new failures still open incidents if a webhook delivery is missed. Every deploy also joins the change ledger. The deployment events Ace records are created, succeeded or ready, error or canceled, promoted, and rollback, and it adds the Git repository, branch and commit when Vercel has them. The webhook delivers every event as it happens, and Ace also polls Vercel for production deploys so they reach the ledger even without the webhook. A deploy belongs to the workload that routes this connection or, if none does, to the workload mapped to the deployed GitHub repository. When a deploy succeeds, Ace rechecks that workload's open incidents, since the release may have fixed them. ## Before you start - The **admin** role in Ace. - A Vercel **Pro** or **Enterprise** team, if you want the account webhook. Without one, skip the webhook and Ace still polls for production deploys. - For actions, a token owner whose team role allows them (see below). ## Credentials and permissions | Credential | Where to create it | Scope or role | Why Ace needs it | | --- | --- | --- | --- | | Access token (needed) | **Account Settings** > **Tokens** | Scope: the team, then All Projects or a single project | Read deployments, build events and projects. Vercel tokens can't be read-only, so the scope is the limit. | | Same token (only for deploy actions) | Same | Owner, Member, or Developer with Full Production Deployment | Rollback, promotion, redeploy, cancel, rolling releases and environment variable changes. | | Same token (only for firewall actions) | Same | Owner, Member or Security | Attack Challenge Mode and firewall IP rules. | | Webhook secret (only for the webhook) | Team **Settings** > **Webhooks**, in the Webhook Created dialog | None | Verifies that deliveries come from Vercel. Vercel shows it only once. | If you enter a **Team ID**, Ace reads the token owner's team role and shows which actions it allows. For other roles, or without a Team ID, Settings shows that the permission is checked when the action runs. ## Set up ### 1. Create a token in Vercel 1. Select your personal account and open **Account Settings** > **Tokens**. 2. Name the token Ace. Under **Scope**, pick the team, then All Projects or a single project. 3. Choose an expiration, create the token and copy it. ### 2. Connect Vercel in Ace 1. Open **Settings** > **Data sources** and choose **Vercel**. 2. Fill in the form: | Field | What to enter | | --- | --- | | **Access token** | The token you created. | | **Team ID** | Only for a Full Account token. Find it under team **Settings** > **General** > **Team ID**. A team- or project-scoped token carries its team. | | **Projects** | Optional. A comma-separated allowlist of project IDs. | 3. Select **Connect Vercel**. Ace checks that it can read your projects before it saves. ### 3. Add the deployment webhook 1. Copy the webhook URL Ace shows. 2. In Vercel, open team **Settings** > **Webhooks** and select the Deployment events: Created, Succeeded, Error, Promoted and Rollback. 3. Choose the projects, paste the URL as the endpoint and select **Create Webhook**. 4. Copy the secret from the Webhook Created dialog, paste it into Ace and select **Save**. ### 4. Optional: allow actions Turn on **Allow Ace to run approved actions**. To let actions earn Automatic, confirm their workload and environment on the same panel. ## Confirm it's working - The connection's checks show how many projects the token can see and name any allowlisted project it can't see. - The webhook badge shows **Not set up**, **Waiting for first event**, **Verified** or **Last delivery failed**. Trigger a deployment, and the badge turns **Verified** when the first signed delivery arrives. - New production deploys appear on the workload's timeline in Ace within a few minutes, even without the webhook. ## Disconnect 1. In Ace, open the Vercel connection under **Settings** > **Data sources** and select **Disconnect**. To keep reading deploys and stop actions, turn off **Allow Ace to run approved actions** instead. 2. In Vercel, delete the Ace token under **Account Settings** > **Tokens** and the webhook under team **Settings** > **Webhooks**. Ace creates nothing else in Vercel. ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals) - [GitHub](https://tryace.dev/docs/integrations/github) - [All integrations](https://tryace.dev/docs/integrations) --- Source: https://tryace.dev/docs/integrations/cloudflare # Cloudflare > Connect Cloudflare so Ace tells edge errors from origin errors, records DNS, rule and Worker changes next to your deploys, opens incidents from Cloudflare notifications, and proposes edge fixes for approval. Source: https://tryace.dev/docs/integrations/cloudflare Ace reads Cloudflare traffic and errors per hostname, security events, Workers and load balancers to investigate incidents at the edge. Cloudflare notifications open incidents, and DNS, rule, SSL and Worker changes join your workloads' change timeline so you see them next to your deploys. | | | | --- | --- | | Category | Edge | | Ace reads | HTTP analytics, firewall events, Workers analytics, audit logs, load balancer health, SSL status, Cloudflare's status page | | Ace can change | Cache purge, Worker rollback, load balancer failover, security level, WAF rule toggle, each after approval | | Opens incidents | Yes, from Cloudflare notifications, for workloads that route Cloudflare | | Setup | Settings form | ## What Ace reads Ace reads every zone the token can see, or only the zones you list. - HTTP analytics per hostname. Ace compares requests and the 5xx share with an equal window before the incident, and splits origin errors (your origin answered 5xx) from edge errors (Cloudflare couldn't reach or talk to the origin) so you know which side failed. - Firewall events from WAF, managed and custom rules, rate limiting and bots. Ace groups them by action, rule, host and path, country, network and user agent, and compares them with the window before. - Workers analytics per script: requests, errors, status codes and CPU time. Ace adds exception messages where it can read Workers Observability. - Audit logs of account changes around the incident, with who made them. These cover DNS, zone and SSL settings, WAF and rate-limit rules, load balancers, pools, monitors, Tunnels, Workers and routes. - Load balancer health: pool and origin health, which pools served traffic over the window, and failovers. - SSL mode and edge certificates, for 525 and 526 errors. - Platform incidents on Cloudflare's status page during the window. Load balancer health and SSL status need the optional read permissions below. Ace checks them when you connect and uses only the reads your token allows. To verify a fix, Ace reads the edge error rate for the affected host. For the System Map, it discovers zones, DNS records, Workers, Worker routes, Tunnels and load balancers. ## What Ace can change Ace proposes these actions once an admin turns on **Allow Ace to run approved actions** for the connection. Each one needs approval by default, and earned autonomy can let some of them run under an Automatic policy for a workload and environment you confirm. Others always need a person. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). | Action | What it does | Approval | Undo | | --- | --- | --- | --- | | Cloudflare cache purge | Purges cached copies of specific URLs, cache tags or URL prefixes in one zone. | Needs approval, can earn Automatic | No. Nothing restores a cache. | | Cloudflare Worker rollback | Deploys an earlier version of a Worker at 100%. | Needs approval, can earn Automatic | Yes. Redeploys the split it replaced. | | Cloudflare load balancer failover | Disables or enables a load balancer pool, or one origin in a pool, to fail traffic over. | Always a person | Yes | | Cloudflare security level | Sets a zone's security level, for example I'm Under Attack. | Always a person | Yes | | Cloudflare WAF rule toggle | Disables or enables one WAF custom rule or rate limiting rule in a zone. | Always a person | Yes | Cloudflare can't report a token's edit permissions, so Settings lists each action as **Shows when it runs: needs ``**, and a missing permission shows up when Cloudflare refuses the change. ## What opens an incident Notifications reach Ace through a webhook destination as they fire, and one opens an incident only when a workload routes Cloudflare. To route it, open the workload's settings, turn on **Integration alerts** under **Investigation triggers**, and check Cloudflare under **Alert sources**. These notification types open an incident, or join the one already open on the workload: | Notification | Severity | | --- | --- | | Origin error rate (`http_alert_origin_error`) | High | | Edge error rate (`http_alert_edge_error`) | High | | Advanced HTTP error alert (`advanced_http_alert_error`) | High | | Passive Origin Monitoring (`real_origin_monitoring`) | Critical | | Load balancing health (`load_balancing_health_alert`) | High | | Health check status (`health_check_status_notification`) | High | | Tunnel health (`tunnel_health_event`) | High | The notification's end event, or a healthy status in it, resolves the incident. Load balancer pools and origins resolve separately. These join an open incident as evidence and never open one: - Cloudflare platform incidents (`incident_alert`). - Custom certificate expiration for hostnames and zones. - A load balancing pool being enabled or disabled. - Maintenance notifications. Universal, dedicated and custom SSL certificate events go to the change timeline. Ace drops every other notification type. Ace also polls Cloudflare, so new failures still open incidents if a notification goes missing. Account changes from the audit log and Worker deployments go to the change timeline. If you list zones by name, Ace drops notifications from other zones. A list of zone IDs can't filter them, because health check notifications name their zone, not its ID. ## Before you start - The **admin** role in Ace. See [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions). - Permission to create API tokens for the Cloudflare account. - The right Cloudflare plan and products for the notifications you want. Error-rate alerts need an Enterprise zone, and health check alerts need Pro or above. Others need Passive Origin Monitoring, Load Balancing, Advanced Certificate Manager or Cloudflare Tunnel. Ace subscribes only to notifications your account has. ## Credentials and permissions Ace uses one custom **API token**, limited to this account and its zones. | Credential | Where to create it | Permissions | Why Ace needs it | | --- | --- | --- | --- | | API token (needed) | `https://dash.cloudflare.com/profile/api-tokens` | Zone: Zone Read, Zone: Analytics Read, Zone: DNS Read, Account: Account Analytics Read, Account: Account Settings Read, Account: Workers Scripts Read | Read zones, traffic, DNS, account analytics, audit logs and Workers. | | Same token (optional reads) | Same | Zone: Load Balancers Read and Account: Load Balancing: Monitors and Pools Read | Load balancer health. | | | | Zone: Workers Routes Read and Account: Cloudflare Tunnel Read | Worker routes and Tunnels on the System Map. | | | | Zone: SSL and Certificates Read and Zone: Zone Settings Read | SSL status for 525 and 526 errors. | | | | Zone: Health Checks Read and Account: Workers Observability Edit | Health checks, and Worker exception messages. Workers Observability has no read level. | | Same token (only for alerts) | Same | Account: Notifications Edit | Add Ace's webhook destination and notifications. | | Same token (only for actions) | Same | Zone: Cache Purge | Cache purge. | | | | Account: Workers Scripts Edit | Worker rollback. | | | | Account: Load Balancing: Monitors and Pools Edit | Load balancer failover. | | | | Zone: Zone Settings Edit | Security level. | | | | Zone: Zone WAF Edit | WAF rule toggle. | ## Set up ### 1. Create the API token 1. In Ace, open **Settings** > **Data sources** and choose **Cloudflare**. Enter your **Account ID**, then select **Create key** next to **API token**. Cloudflare's token page opens with a custom token named Ace and the six required read permissions added. 2. Add the optional read rows you want. Add Account: Notifications Edit if Ace should wire alerts. 3. Add edit rows only for the actions Ace may take after approval. 4. Under **Account Resources**, include only this account. Under **Zone Resources**, include the zones you'll list in Ace, or all zones of the account. 5. Optionally set a TTL. Select **Continue to summary**, then **Create Token**, and copy the token. ### 2. Connect Cloudflare in Ace 1. Fill in the form: - **Account ID**: the 32-character ID. On Account home, open Search (Cmd/Ctrl+K) and choose **Copy account ID**. You can also copy **Account ID** under **Account Details** on the Workers & Pages page. - **API token**: the token you copied. - **Zones** (optional): zone names or IDs, comma-separated. Use names if Ace wires alerts. 2. Select **Connect Cloudflare**. Ace saves only if the token passes validation. ### 3. Wire alerts If the token has Notifications Edit, Ace creates a webhook destination, attaches the health check, load balancing, Tunnel, SSL, error-rate and Cloudflare incident notifications your account has, and reports what it set up in the **Alerts** section. Ace never changes your own notifications, and it rechecks its setup hourly and removes it when you disconnect. To wire alerts yourself, select **Show manual setup** in the **Alerts** section: 1. Copy the webhook URL. 2. Select **Generate token** and copy the token. Ace shows it once. 3. In Cloudflare, open **Notifications** > **Destinations**, add a webhook with the URL, and paste the token as its **Secret**. Cloudflare sends it in the `cf-webhook-auth` header of every notification. 4. Create the notifications that should open incidents, such as Health Checks, Load Balancing Health, Tunnel Health, SSL and Origin Error Rate, and send them to this webhook. ### 4. Optional: allow actions 1. On the connected Cloudflare page, turn on **Allow Ace to run approved actions**. 2. For automatic actions, select a workload, enter its environment (default `production`), and select **Confirm**. ## Confirm it's working - The connection shows **Connected** once Ace verifies that the token can read your zones, analytics, DNS, audit logs and Workers. - The **Alerts** section shows **Waiting for first event** until Ace accepts a notification, then **Verified**. ## Disconnect 1. In Ace, open **Settings** > **Data sources** > **Cloudflare** and select **Disconnect**. Ace removes its notifications and webhook destination, and if it can't, it tells you so you can remove `ace-` in Cloudflare yourself. 2. In Cloudflare, delete the API token. ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals) - [Operating rules](https://tryace.dev/docs/concepts/operating-rules) - [All integrations](https://tryace.dev/docs/integrations) --- Source: https://tryace.dev/docs/integrations/trigger-dev # Trigger.dev > Connect Trigger.dev so failed production runs open incidents, deploys join the change ledger, and Ace can replay or cancel runs and pause queues after approval. Source: https://tryace.dev/docs/integrations/trigger-dev Failed Trigger.dev runs open incidents in Ace, one incident per task. Once the fix is live, Ace can replay the failed runs in stages. | | | | --- | --- | | Category | Jobs | | Ace reads | Failed runs, run details, traces and logs, error groups, queues, schedules, deployed version, compute cost of failed runs | | Ace can change | Replay failed runs, cancel runs, pause or resume a queue, only after an admin allows actions | | Opens incidents | Yes, for failed production runs of workloads that route Trigger.dev | | Setup | Settings form, plus a Trigger.dev alert webhook | ## What Ace reads Ace reads Trigger.dev while it investigates a workload that lists it under **Investigation sources** or routes its alerts. From your production environment it reads: - Runs that ended `FAILED`, `CRASHED`, `SYSTEM_FAILURE` or `TIMED_OUT` in the incident window, grouped by task and status, with versions and a sample run. - One run's status, error, stack frames, attempts and version. Ace doesn't get the payload. - Error groups, run traces and run logs. - Read-only queries over runs. - Queue depth and whether a queue is paused. - Failed runs of the incident's task after a fix, plus at least one completed run, to confirm the task works again. Ace always leaves out test runs, and a **Tasks** allowlist limits reads to those tasks. For the System Map, Ace reads tasks, queues, schedules and the deployed version. An Observer key can read queues, but schedules and deployments need a broader preset and show as not permitted without one. A scheduled cost scan reports the compute spent on failed runs, though Ace can't change that cost for you. ## What Ace can change Ace proposes Trigger.dev actions only after an admin turns on **Allow Ace to run approved actions** for the connection. Each one needs approval by default, and earned autonomy can let some of them run under an Automatic policy for a workload and environment you confirm. Others always need a person. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). | Action | What it does | Approval | Undo | | --- | --- | --- | --- | | Run replay | Replays the incident's failed runs in stages, for example 10 first and then the rest, up to 2,000 runs. It waits for the incident's fix to deploy before it starts. | Approval, or Automatic under your policy | No | | Run cancel | Cancels a runaway task's queued and executing runs in stages, up to 2,000 runs. | Always a person | No. A cancelled run can't be resumed. | | Queue pause | Pauses or resumes a queue. | Always a person | Yes. Restores the queue's previous state. | Ace applies these safeguards: - The replay list is fixed when the first stage starts, and each run replays at most once. Ace never replays its own new runs. - The cancel list is fixed at approval. - Ace refuses tasks outside your **Tasks** allowlist, and it refuses to pause or resume a queue that changed since approval. - Ace rechecks the incident between stages. These actions need the **Operator** key preset. Trigger.dev doesn't let Ace check the preset in advance, so Settings shows that the permission is checked when the action runs. ## What opens an incident Trigger.dev alerts open incidents only on workloads that route this connection (**Integration alerts** on, Trigger.dev selected). Ace drops alerts that no workload routes. | Trigger.dev sends | What Ace does | | --- | --- | | A failed run alert | Opens or joins an incident for that task. `SYSTEM_FAILURE` and `CRASHED` are critical, and other failures are high. | | A new or regressed error group | Opens or joins an incident. It carries the same task and error as the failed runs, so it joins their incident. | | An error group that was unignored | Adds context to an open incident. It never opens one. | | A deployment that succeeded or failed | Joins the change ledger, never an incident. Ace records the commit when Trigger.dev has git metadata for the deploy. | Only production failures open incidents, while failures in staging, preview or development add context to an open incident. Ace ignores test runs. Ace also polls for failed runs, so failures still open incidents before you set up the webhook or when a delivery goes missing. Polling doesn't read deployments, which arrive only through the webhook. ## Before you start - The **admin** role in Ace. - A key for your **Production** environment, which starts with `tr_prod_`. - A Trigger.dev **Pro** or **Enterprise** plan for the **Observer** and **Operator** key presets. Other plans and self-hosted Trigger.dev offer only **No restrictions**. ## Credentials and permissions | Credential | Where to create it | Preset | Why Ace needs it | | --- | --- | --- | --- | | Secret key (needed) | Project > **Production** environment > **API keys** > **New API key** | Observer | Read runs, errors, traces, logs and queues. | | Same key (only for actions) | Same | Operator | Replay and cancel runs, and pause or resume queues. | | Webhook secret (only for alerts) | Shown when you create the alert in Trigger.dev | None | Verifies that alert deliveries come from Trigger.dev. | ## Set up ### 1. Create an API key in Trigger.dev 1. Pick the project, switch to the **Production** environment, and open **API keys** > **New API key**. 2. Name it Ace and set **Expires** (In 90 days or In 1 year). 3. Choose **Observer**, or **Operator** if Ace should replay runs. 4. Create the key and copy it, because Trigger.dev shows it only once. ### 2. Connect Trigger.dev in Ace 1. Open **Settings** > **Data sources** and choose **Trigger.dev**. 2. Fill in the form: | Field | What to enter | | --- | --- | | **Secret key** | The production key (`tr_prod_…`). | | **API URL** | Only for self-hosted Trigger.dev. Leave blank for `https://api.trigger.dev`. | | **Tasks** | Optional. A comma-separated allowlist of task identifiers. | 3. Select **Connect Trigger.dev**. Ace checks that the key can read runs before it saves. A self-hosted **API URL** must use HTTPS and be reachable from the public internet, because Ace doesn't connect to private or internal network addresses. ### 3. Send alerts to Ace 1. In the **Alerts** section, copy the webhook URL. 2. In your Trigger.dev project, open **Alerts**, choose **New alert** and pick **Webhook**. 3. Under **Alert me when**, select **Task runs fail**, **Deployments fail** and **Deployments succeed**. Paste the URL and create the alert. 4. Optional: on the **Errors** page, choose **Configure alerts…** and add the same webhook. New and regressed errors then join the incident of the runs they fail. 5. Copy the webhook secret Trigger.dev shows, paste it into Ace and select **Save**. ### 4. Route alerts to a workload Open the workload in Ace, turn on **Integration alerts**, and select Trigger.dev as a source. ### 5. Optional: allow actions Turn on **Allow Ace to run approved actions**. To let replays earn Automatic, confirm their workload and environment on the same panel. ## Confirm it's working - If the key can't read runs, the connection's checks report missing Observer access. - The **Alerts** badge shows **Not set up**, **Waiting for first event**, **Verified** or **Last delivery failed**. It turns **Verified** when the first signed delivery arrives. - A failed delivery shows the reason, such as a signature that doesn't match the saved secret. ## Disconnect 1. In Ace, open the Trigger.dev connection under **Settings** > **Data sources** and select **Disconnect**. To stop actions and keep the connection, turn off **Allow Ace to run approved actions** instead. 2. In Trigger.dev, delete the Ace API key and remove the alert webhook. Ace creates nothing in Trigger.dev. ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals) - [Inngest](https://tryace.dev/docs/integrations/inngest) - [All integrations](https://tryace.dev/docs/integrations) --- Source: https://tryace.dev/docs/integrations/inngest # Inngest > Connect Inngest so failed production runs open incidents, app syncs join the change ledger, and Ace can rerun, cancel or pause after approval. Source: https://tryace.dev/docs/integrations/inngest Failed Inngest runs open incidents in Ace, one incident per function. Once the fix is live, Ace can rerun the failed runs in stages. | | | | --- | --- | | Category | Jobs | | Ace reads | Failed runs, run details and step traces, apps and functions, backlog, step failures from Insights | | Ace can change | Rerun failed runs, cancel runs, pause or resume a function, only after an admin allows actions | | Opens incidents | Yes, for failed production runs of workloads that route Inngest | | Setup | Settings form, plus an optional forwarding function in your app | ## What Ace reads Ace reads Inngest while it investigates a workload that lists it under **Investigation sources** or routes its alerts. Ace reads only the environment named in the **Environment** field and sends it with every request, so a key that covers several environments still reads only this one. An **Apps** allowlist limits reads to those apps. During an investigation Ace reads: - Failed runs in the incident window, grouped by function and error, with triggering events, sample run IDs, error messages and in-app stack frames. - One run's status, timing, trigger, error and step trace. Ace doesn't get inputs or outputs. - Apps with their latest sync, and each function's triggers, paused state, retries and flow control. - The backlog of queued, running and paused runs since the incident began, by function. - Failed step attempts by function and step, from Inngest Insights, whose data lags a few minutes. - Failed runs of the incident's function after a fix, plus at least one completed run, to confirm the function works again. Ace also adds your apps and functions to the System Map. Ace blocks any Inngest request beyond these reads and the actions below before it leaves Ace. ## What Ace can change Ace proposes Inngest actions only after an admin turns on **Allow Ace to run approved actions** for the connection. Each one needs approval by default, and earned autonomy can let some of them run under an Automatic policy for a workload and environment you confirm. Others always need a person. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). | Action | What it does | Approval | Undo | | --- | --- | --- | --- | | Run rerun | Reruns the failed runs of one function from the incident window in stages, for example 10 first and then the rest, up to 500 runs. Each run reruns exactly once. It waits for the incident's fix to deploy before it starts. | Approval, or Automatic under your policy | No | | Function pause | Pauses one function so no new runs start. Use it as a kill switch. Inngest cancels runs still unfinished after 7 days paused. | Always a person | Yes. Resumes the function. | | Function resume | Resumes a paused function. | Always a person | Yes | | Run cancel | Cancels one function's queued, running and paused runs from the incident window in stages, up to 500 runs. Each run is cancelled exactly once. | Always a person | No. A cancelled run can't be resumed. | A rerun replays every step from the original event, so non-idempotent side effects happen again. Ace rechecks the incident between stages. These actions need an API key with **Write** permission. Inngest doesn't let Ace check this in advance, so Settings shows that the permission is checked when the action runs. ## What opens an incident Failures open incidents only on workloads that route this connection (**Integration alerts** on, Inngest selected). Ace polls only routed connections. - Ace polls every few minutes for runs with status `FAILED`. Each failure opens or joins an incident for its function, at high severity. - For alerts within seconds, forward Inngest's `inngest/function.failed` event to Ace (step 4 under [Set up](#set-up)). Ace treats a forwarded failure the same as a polled failure of that run. - Failures open incidents only when the connected environment is production. In any other environment they add context to an open incident. App syncs join the change ledger as deploys, and function pauses and resumes join it as configuration changes, dated when Ace first sees them. Neither opens an incident. ## Before you start - The **admin** role in Ace. - An Inngest **organization admin**, to create API keys. - An API key (`sk-inn-api-…`). Ace refuses signing keys, because a signing key is your app's own secret and can do everything in the environment. ## Credentials and permissions | Credential | Where to create it | Permission | Why Ace needs it | | --- | --- | --- | --- | | API key (needed) | Profile menu > **API Keys** | Read, limited to the environment Ace reads | Read runs, traces, apps, functions and Insights. | | Same key (only for actions) | Same | Write | Rerun and cancel runs, and pause or resume functions. | | Token (only for forwarded failures) | Generated by Ace | None | Proves that a forwarded failure comes from your app. | ## Set up ### 1. Create an API key in Inngest 1. Open the profile menu > **API Keys**. 2. Create a key named Ace with **Read** permission, or **Write** if Ace should rerun, cancel and pause after approval. Limit the key to the environment Ace reads, which is production unless you name another. 3. Copy the key (`sk-inn-api-…`). ### 2. Connect Inngest in Ace 1. Open **Settings** > **Data sources** and choose **Inngest**. 2. Fill in the form: | Field | What to enter | | --- | --- | | **API key** | The key you created. | | **Environment** | Optional. The environment slug Ace reads. Defaults to `production`. | | **Apps** | Optional. A comma-separated allowlist of app IDs (the `id` you pass to `new Inngest({ id })`). | 3. Select **Connect Inngest**. Before it saves, Ace checks that the key can read runs and apps in that environment. ### 3. Route failures to a workload Open the workload in Ace, turn on **Integration alerts**, and select Inngest as a source. Polling starts within a few minutes. ### 4. Optional: forward failures for faster alerts 1. In the **Alerts** section, copy the webhook URL. 2. Select **Generate token** and copy the token, which Ace shows only once. Store the URL and token in your app's environment, for example as `ACE_INNGEST_URL` and `ACE_INNGEST_TOKEN`. 3. Add a function triggered by `inngest/function.failed` to an app in the same environment. It sends the event unchanged as JSON, inside `step.run` so Inngest retries it. This example uses the Inngest TypeScript SDK v4. On SDK v3, pass `{ event: "inngest/function.failed" }` as a separate second argument instead of `triggers`. ```ts title="forward-failures-to-ace.ts" { id: "forward-failures-to-ace", triggers: [{ event: "inngest/function.failed" }] }, async ({ event, step }) => { await step.run("send-to-ace", async () => { const response = await fetch(process.env.ACE_INNGEST_URL!, { method: "POST", headers: { "content-type": "application/json", authorization: `Bearer ${process.env.ACE_INNGEST_TOKEN}`, }, body: JSON.stringify(event), }); if (!response.ok) throw new Error(`Ace returned ${response.status}`); }); }, ); ``` Ace accepts a forwarded failure only if its function and run IDs have Inngest's format and the function is in your **Apps** allowlist. ### 5. Optional: allow actions Turn on **Allow Ace to run approved actions**. To let reruns earn Automatic, confirm their workload and environment on the same panel. ## Confirm it's working - The connection's checks show which environment the key reads and how many apps it sees, and flag an environment that doesn't exist for the key, or an allowlisted app it can't see. - Once forwarding is set up, the **Alerts** badge shows **Not set up**, **Waiting for first event**, **Verified** or **Last delivery failed**. It turns **Verified** when the first delivery with the right token arrives. - To rotate the token, select **Generate a new token** and update your app, since the old token stops working. ## Disconnect 1. In Ace, open the Inngest connection under **Settings** > **Data sources** and select **Disconnect**. To stop actions and keep the connection, turn off **Allow Ace to run approved actions** instead. 2. In Inngest, delete the Ace API key, and remove the forwarding function from your app if you added it. Ace creates nothing in Inngest. ## Next steps - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals) - [Trigger.dev](https://tryace.dev/docs/integrations/trigger-dev) - [All integrations](https://tryace.dev/docs/integrations) --- Source: https://tryace.dev/docs/integrations/supabase # Supabase > Connect Supabase so unhealthy services, Supabase health checks, read-only mode and a full disk open incidents, and Ace can fix pool size, disk and read-only mode after approval. Source: https://tryace.dev/docs/integrations/supabase Ace watches Supabase project health and investigates database incidents through the Management API. It never runs SQL and never sees your table data. | | | | --- | --- | | Category | Database | | Ace reads | Project and service health, metrics, logs, advisors, disk, configuration, usage, edge function deploys, migrations, branches | | Ace can change | Pooler pool size, a temporary lift of read-only mode, disk size, project restart. A person always approves. | | Opens incidents | Yes, for workloads that route Supabase | | Setup | Settings form | ## What Ace reads Ace reads Supabase while it investigates a workload that lists it under **Investigation sources** or routes its alerts. It reads only the projects you list. - Project status and the health of Postgres, logins, the connection pooler, the dedicated pooler, Auth, the Data API, Realtime and Storage. Ace also reads read-only mode, disk use, `max_connections` and `statement_timeout`. - A current metrics snapshot: Postgres up, CPU per core, memory, disk, I/O, backends, pooler connections and replication lag. - One log source at a time over the incident window, for example edge, Postgres, Auth or edge functions. - Performance health checks and security advisors, errors first. - Request counts per service and edge function invocation stats. Supabase keeps 7 days of these. - Edge function deploys, migration versions and branch action runs around the incident. For the System Map, Ace reads projects, the connection pooler, custom domains and branches. If the token lacks an optional permission, Ace skips the reads that need it and uses the rest. ### What Ace never reads Ace can send only the Supabase requests listed above. It blocks the SQL endpoint, function code, API keys and every write other than the four actions below before the request leaves Ace. Ace never needs the **Database** permission, so don't grant it. ## What Ace can change Every Supabase action needs a person's approval and never runs automatically, whatever your policy says. Ace proposes them only after an admin turns on **Allow Ace to run approved actions** for the connection, and only where the token has Read-write. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). | Action | What it does | Permission | Undo | | --- | --- | --- | --- | | Pooler pool size | Changes the connection pooler's default pool size (1 to 3,000), and optionally its pool mode. | Connection Pooling: Read-write | Yes. Restores the previous settings. | | Read-only mode lift | Lifts read-only mode for 15 minutes after the disk filled, so writes can free space. It ends on its own. | Read-only Mode: Read-write | No | | Disk increase | Grows the database disk. A disk can never shrink, and Supabase makes the next disk change wait about 4 hours. | Disk Config: Read-write | No | | Project restart | Restarts the project. Postgres and its services restart, and every client reconnects. | Project Settings: Read-write | No | Ace doesn't change the pooler's max client connections or restart single services, and it refuses to write to any project outside the connection. Supabase can't report a token's permissions, so Settings shows that each write permission is checked when the action runs. Pool size is the exception, because if the token can't read the pooler, Ace knows it can't change it either. > **Project Settings: Read-write** > > Read-write on Project Settings lets Ace restart the project, and in Supabase it also allows pausing or deleting it. Ace never sends those requests. Grant read-write only if you want Ace to propose restarts. ## What opens an incident Problems open incidents only on workloads that route this connection (**Integration alerts** on, Supabase selected). Supabase has no webhook, so Ace checks routed projects on a schedule. An incident opens while a problem is present and resolves when it clears. | Condition | Severity | | --- | --- | | Postgres unhealthy | Critical | | Auth or Data API unhealthy | High | | Realtime unhealthy | Medium | | Read-only mode on | Critical | | Disk nearly full | High, or critical when the disk is close to full | | Supabase advisor health check at level ERROR or WARN | Depends on the check. WARN is one step lower. | These advisor health checks open incidents: - Critical: database down, database not reachable, database connections failing. - High: connection limit reached, an instance alert firing, high Data API or Auth error rate. - Medium: high Storage or edge function error rate, project not active. These add context and never open an incident: - Unhealthy Postgres logins, connection pooler, dedicated pooler or Storage. - Project status, such as unhealthy, restoring, inactive, or a failed restore or pause. If Ace loses read access to a project, its open problems expire after a while, so a lost permission can't hold an incident open. Edge function deploys, migrations and branch action runs (migrate and deploy steps) join the change ledger whether or not a workload routes the connection, and they never open incidents. Supabase doesn't report when a migration was applied, so Ace dates it when it first sees it. ## Before you start - The **admin** role in Ace. - A **scoped** personal access token (`sbp_fc…`). Ace refuses Classic (Legacy) tokens, which carry your whole account. - The Project ID of each project Ace should read. It's the 20-letter ID under **Project Settings** > **General** > **Project ID**, and also the part after `/project/` in the dashboard URL. ## Credentials and permissions Ace uses one scoped access token, limited to the projects you list. Don't use the **Read-only** preset, which also grants Database and secret reads. | Permission | Needed for | Why Ace needs it | | --- | --- | --- | | Project Settings: Read | Needed | Project status and service health. | | Logs: Read | Needed | Logs and metrics. | | Advisors: Read | Needed | Performance health checks and security advisors. | | Disk Config: Read | Needed | Disk use. | | Read-only Mode: Read | Needed | Read-only mode. | | Usage Analytics, Action Runs, Migrations, Database Config, Connection Pooling, Edge Functions, Development Branches: Read | Optional | Traffic, recent changes, configuration and the System Map. Edge Functions: Read also allows reading function code, which Ace never requests. | | Connection Pooling: Read-write | Only for actions | Change the pool size. | | Read-only Mode: Read-write | Only for actions | Lift read-only mode after a full disk. | | Disk Config: Read-write | Only for actions | Grow the disk. | | Project Settings: Read-write | Only for actions | Restart the project. Also allows pausing or deleting it. | | Database | Never | Ace never queries your data. | ## Set up ### 1. Create a scoped access token in Supabase 1. Open **Account** > **Access Tokens** and choose **Generate new token**. Don't use **Create legacy token**. 2. Name it Ace and set **Expires in** (90 days, or a custom date up to a year). 3. Under **Resource access**, choose **Project**, then the organization, then the projects Ace should read. 4. Under **Permissions**, keep **Preset** at No access. Set Read on Project Settings, Logs, Advisors, Read-only Mode and Disk Config. Optionally set Read on Usage Analytics, Action Runs, Migrations, Database Config, Connection Pooling, Edge Functions and Development Branches. Leave every other row at None, especially Database. 5. Only for approved changes, set Read-write on Connection Pooling, Read-only Mode and Disk Config. Set it on Project Settings only if Ace may restart the project. Leave Database at None. 6. Review access, select **Create token**, and copy the `sbp_fc…` token. ### 2. Connect Supabase in Ace 1. Open **Settings** > **Data sources** and choose **Supabase**. 2. Fill in the form: | Field | What to enter | | --- | --- | | **Scoped access token** | The `sbp_fc…` token. | | **Project IDs** | A comma-separated list of the Project IDs Ace reads. Ace reads no other project. | 3. Select **Connect Supabase**. Ace checks the token before it saves, and it never sends a classic token to Supabase. ### 3. Route alerts to a workload Open the workload in Ace, turn on **Integration alerts**, and select Supabase as a source. ### 4. Optional: allow actions Turn on **Allow Ace to run approved actions**. ## Confirm it's working The connection runs the checks `scoped_token`, `project_settings_read`, `logs_read`, `advisors_read`, `disk_config_read` and `read_only_mode_read`. **Re-validate** reruns them. These failures come up most often: - A classic token: "This is a classic (Legacy) token with access to your whole account. Create a scoped token (sbp_fc…) limited to these projects." - A listed project the token can't see, which the check names before asking you to scope the token to it with Project Settings: Read. - No readable project: "No Supabase project could be read." ## Disconnect 1. In Ace, open the Supabase connection under **Settings** > **Data sources** and select **Disconnect**. To stop actions and keep the connection, turn off **Allow Ace to run approved actions** instead. 2. In Supabase, delete the Ace token under **Account** > **Access Tokens**. Ace creates nothing in Supabase. ## Next steps - [PlanetScale](https://tryace.dev/docs/integrations/planetscale) - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals) - [All integrations](https://tryace.dev/docs/integrations) --- Source: https://tryace.dev/docs/integrations/planetscale # PlanetScale > Connect PlanetScale so Ace reads your schema, deploy requests, Query Insights and bounded row counts, and can propose schema changes and backfills that a person approves. Source: https://tryace.dev/docs/integrations/planetscale Ace reads PlanetScale as evidence during investigations. It looks at the schema of the tables an error points at, recent schema changes, slow or heavy query patterns, and row counts that test a hypothesis. | | | | --- | --- | | Category | Database | | Ace reads | Schema, deploy requests, Query Insights, databases and branches, and read-only SQL when you add a read-only password | | Ace can change | Schema changes through deploy requests, and bounded backfills when you add a write password. A person always approves. | | Opens incidents | No. Ace reads PlanetScale during investigations. | | Setup | Settings form | ## What Ace reads Ace reads PlanetScale while it investigates a workload that lists it under **Investigation sources** or routes its alerts. It reads one database and its production branch, `main` unless you set another. The service token lets Ace read: - `CREATE TABLE` statements of the production branch. - Recent deploy requests with their state and deploy time. - Query Insights patterns for the production branch, normalized so they never include literal values. Query Insights must be on for the database. If you add the optional read-only SQL password, Ace runs one `SELECT`, `SHOW`, `DESCRIBE` or `EXPLAIN` statement per request and gets at most 100 rows back. It refuses multiple statements and statements that write files, lock rows or stall the database. The password itself is what stops writes. Ace checks that it has PlanetScale's reader role on the production branch, so the database refuses any write sent with it. For the System Map, Ace reads your databases and branches. ## What Ace can change Every PlanetScale action needs a person's approval and never runs automatically, whatever your policy says. Ace proposes them only after an admin turns on **Allow Ace to run approved actions** for the connection. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). | Action | What it does | Needs | Undo | | --- | --- | --- | --- | | Schema change | Opens a deploy request from an existing development branch into the production branch, then deploys it. | Service token with `create_deploy_request` | Yes, during PlanetScale's revert window | | Schema revert | Reverts a deploy request inside PlanetScale's revert window. | Service token with `create_deploy_request` | No | | Backfill | Updates matching rows in batches, after a dry-run count. | Write SQL password, plus the read-only SQL host, username and password | No | ### How backfills work Ace builds the backfill `UPDATE` from a table name, 1 to 10 columns to set, and 1 to 10 conditions (`IS NULL` or equals). It validates column names and escapes values, and it never runs free-form SQL. 1. **Dry run.** Ace counts matching rows with the read-only password. If none match, or more match than the limit you approve, Ace blocks the backfill. 2. **Approval.** You see the exact `UPDATE` and row count, and you set **Rows per batch** (1 to 1,000), **Most rows to update** (1 to 100,000) and the wait between stages. 3. **Batches.** Ace runs the update with the write password, one batch at a time. One condition must be on a column the backfill sets, so updated rows stop matching and each batch reaches new rows. A backfill can't be undone, and it can't run without a write SQL password. Ace doesn't check that password's role, so give it only the access backfills need. ## What opens an incident Nothing. PlanetScale has no webhook, and Ace doesn't poll it. To use it, add it to a workload's **Investigation sources**. Schema changes made outside Ace don't join the change ledger. Ace reads recent deploy requests during an investigation instead. ## Before you start - The **admin** role in Ace. - Permission to create service tokens and passwords for the database in PlanetScale. - Query Insights turned on for the database, if you want Ace to read it. ## Credentials and permissions | Credential | Where to create it | Permissions | Why Ace needs it | | --- | --- | --- | --- | | Service token ID and service token (needed) | **Settings** > **Service tokens** | `read_database`, `read_branch`, `read_deploy_request` on the database | Read the database, schema, deploy requests and Query Insights. | | Same token (only for schema changes) | Same | `create_deploy_request` | Open, deploy and revert deploy requests. | | Read-only SQL host, username and password (optional) | The database's **Settings** > **Passwords** > **New password**, on the production branch | Role **Read-only** | Read-only SQL and row counts. The backfill dry run also needs it. | | Write SQL username and password (only for backfills) | Same | A role that can update the tables you backfill | Run approved, bounded backfills. | Ace doesn't detect `create_deploy_request` in advance, so Settings shows that the permission is checked when the action runs. ## Set up ### 1. Create a service token in PlanetScale 1. Open **Settings** > **Service tokens** and choose **New service token**. Name it Ace and select **Create service token**. 2. Copy the token ID and the token right away, because PlanetScale can't show either one again. 3. Select **Edit token permissions** > **Add database access** and pick the database. Tick `read_database`, `read_branch` and `read_deploy_request`, plus `create_deploy_request` for schema changes. Select **Save permissions**. ### 2. Optional: create SQL passwords 1. In the database's **Settings** > **Passwords**, choose **New password** on the production branch with role **Read-only**. Copy the host, username and password. PlanetScale shows the password once. 2. Only for backfills, create a second password with write access on the same branch. Copy its username and password. ### 3. Connect PlanetScale in Ace 1. Open **Settings** > **Data sources** and choose **PlanetScale**. 2. Fill in the form: | Field | What to enter | | --- | --- | | **Service token ID** | The token ID. | | **Service token** | The token. | | **Organization** | Your PlanetScale organization name. | | **Database** | The database name. | | **Production branch** | Optional. Defaults to `main`. | | **Read-only SQL host** | Optional. The host from the read-only password. It ends in `.psdb.cloud`. | | **Read-only SQL username** | Optional. | | **Read-only SQL password** | Optional. | | **Write SQL username** | Optional. Only for approved, bounded backfills. | | **Write SQL password** | Optional. | 3. Select **Connect PlanetScale**. Before it saves, Ace checks the token and, if you entered a read-only password, checks that the password works and has the reader role. ### 4. Add PlanetScale to a workload Open the workload in Ace and add PlanetScale under **Investigation sources**. ### 5. Optional: allow actions Turn on **Allow Ace to run approved actions**. ## Confirm it's working The connection runs `database_read`, `schema_read` and `deploy_requests_read` for the service token. For a read-only password it also runs `sql_read` and `sql_read_role`. A password with another role fails with "The read password has the `` role. Use a password with the reader role for read queries." **Re-validate** reruns the checks. ## Disconnect 1. In Ace, open the PlanetScale connection under **Settings** > **Data sources** and select **Disconnect**. To stop actions and keep the connection, turn off **Allow Ace to run approved actions** instead. 2. In PlanetScale, delete the Ace service token and any SQL passwords you created for Ace. Ace creates nothing else in PlanetScale. ## Next steps - [Supabase](https://tryace.dev/docs/integrations/supabase) - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals) - [All integrations](https://tryace.dev/docs/integrations) --- Source: https://tryace.dev/docs/integrations/posthog # PostHog > Connect PostHog so insight alerts open incidents, Ace measures who is affected, and Ace can roll a feature flag out in stages or switch it off after approval. Source: https://tryace.dev/docs/integrations/posthog Ace reads PostHog to measure product impact during an incident, finding which events changed, how many users are affected and which feature flags are involved. Insight alerts can open incidents, and Ace can roll a flag out or switch it off as a kill switch. | | | | --- | --- | | Category | Product | | Ace reads | Event volumes and trends, distinct users, feature flags, exceptions, annotations | | Ace can change | Feature flag rollout percentage and on/off state, only after an admin allows actions | | Opens incidents | Yes, from insight alerts, for workloads that route PostHog | | Setup | Settings form, plus a webhook destination on a PostHog insight alert | ## What Ace reads Ace reads one PostHog project while it investigates a workload that lists PostHog under **Investigation sources** or routes its alerts. - Events whose volume changed most in the incident window, compared with an equal window before it. - 15-minute counts and distinct users for one event. - Flags that match a search, with their active state and rollout percentage. - Exceptions in the window, grouped by error tracking issue, with type, message, event count and affected users. - Your team's annotations around the incident. - The incident's failure event after a fix, to confirm the problem stopped. Ace also adds the connected project to the System Map. ## What Ace can change Ace proposes PostHog actions only after an admin turns on **Allow Ace to run approved actions** for the connection. Each one needs approval by default, and earned autonomy can let either one run under an Automatic policy for a workload and environment you confirm. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). | Action | What it does | Approval | Undo | | --- | --- | --- | --- | | Flag rollout | Rolls a flag out in stages, for example 10%, 50%, then 100%. It changes the rollout percentage of one release condition only. You set the wait between stages (60 to 3,600 seconds). | Approval, or Automatic under your policy | Yes. Restores the previous rollout. | | Flag switch | Turns a flag off as a kill switch, or turns it on. | Approval, or Automatic under your policy | Yes. Restores the previous state. | Ace applies these safeguards: - Ace writes with the flag version it saw at approval, so PostHog refuses the change if anyone edited the flag since. - PostHog also refuses a switch when the flag needs a PostHog approval or other active flags depend on it. - Ace rechecks the incident between stages. These actions need the `feature_flag:write` scope, and Settings shows whether the key has it. ## What opens an incident PostHog alerts open incidents only on workloads that route this connection (**Integration alerts** on, PostHog selected). Ace accepts insight alerts sent to a webhook destination and drops alerts that no workload routes. - An `$insight_alert_firing` event opens or joins an incident at high severity. Each alert is its own group, so repeated firings of one alert join one incident. - An alert that reports "not firing", "resolved" or "ok" resolves it. - Ace ignores other events, and it accepts a custom body only if the body names an alert ID. Ace doesn't poll PostHog, so without the webhook PostHog alerts don't reach Ace. ## Before you start - The **admin** role in Ace. - A PostHog insight with an alert, or permission to create one. - Your **Project ID**, under **Settings** > **General** in **Project token & ID**. ## Credentials and permissions Limit the key to this one project. Don't use the **Read-only access** preset, which grants far more than Ace needs. | Credential | Where to create it | Scopes | Why Ace needs it | | --- | --- | --- | --- | | Personal API key (needed) | **Settings** > **Personal API keys** | Read on Project, Query, Feature flag and Annotation (`project:read`, `query:read`, `feature_flag:read`, `annotation:read`) | Read the project, run queries, and read flags and annotations. | | Same key (only for actions) | Same | Write on Feature flag (`feature_flag:write`) | Roll flags out and switch them on or off. | | Token (only for alerts) | Generated by Ace | None | Proves that alert deliveries come from your PostHog webhook destination. | ## Set up ### 1. Create a personal API key in PostHog 1. Open **Settings** > **Personal API keys** and choose **Create personal API key**. 2. Label it Ace and set **Expiration**, for example 90 days. 3. Under **Organization & project access**, choose **Projects** and select only this project. 4. Leave **Select preset** alone. Under **Scopes**, set Read on Project, Query, Feature flag and Annotation. Set Write on Feature flag only if Ace should roll out or switch flags. 5. Create the key and copy it. ### 2. Connect PostHog in Ace 1. Open **Settings** > **Data sources** and choose **PostHog**. 2. Fill in the form: | Field | What to enter | | --- | --- | | **Personal API key** | The key you created. | | **PostHog URL** | Optional. Leave blank for `https://us.posthog.com`. Use `https://eu.posthog.com` for the EU cloud, or your own host. | | **Project ID** | The numeric project ID. | 3. Select **Connect PostHog**. Before it saves, Ace checks that the key can read the project, run a query and read feature flags. A self-hosted **PostHog URL** must use HTTPS and be reachable from the public internet, because Ace doesn't connect to private or internal network addresses. ### 3. Send alerts to Ace 1. In the **Alerts** section, copy the webhook URL. 2. Select **Generate token** and copy the token, which Ace shows only once. 3. In PostHog, open the insight to watch, open **Alerts**, create an alert and add a **Webhook** destination. 4. Paste the URL and keep the default JSON body. Under **Secret headers**, add `Authorization` with the value `Bearer`, a space and the token. ### 4. Route alerts to a workload Open the workload in Ace, turn on **Integration alerts**, and select PostHog as a source. ### 5. Optional: allow actions Turn on **Allow Ace to run approved actions**. To let flag actions earn Automatic, confirm their workload and environment on the same panel. ## Confirm it's working - If the host is wrong for your region, the connection's checks suggest the US or EU host. - The **Alerts** badge shows **Not set up**, **Waiting for first event**, **Verified** or **Last delivery failed**. Trigger the alert or send a test from PostHog, and the badge turns **Verified** when the first delivery with the right token arrives. - A failed delivery shows the reason, such as a bearer token that doesn't match. - To rotate the token, select **Generate a new token** and update the PostHog secret header. ## Disconnect 1. In Ace, open the PostHog connection under **Settings** > **Data sources** and select **Disconnect**. To stop actions and keep the connection, turn off **Allow Ace to run approved actions** instead. 2. In PostHog, remove the webhook destination from your alerts and delete the Ace personal API key. Ace creates nothing in PostHog. ## Next steps - [LaunchDarkly](https://tryace.dev/docs/integrations/launchdarkly) - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals) - [All integrations](https://tryace.dev/docs/integrations) --- Source: https://tryace.dev/docs/integrations/launchdarkly # LaunchDarkly > Connect LaunchDarkly so flag changes join the change ledger next to deploys, and Ace can propose a kill switch, a staged rollout or a version restore. Source: https://tryace.dev/docs/integrations/launchdarkly Every LaunchDarkly flag and segment change joins the change ledger, Ace's per-workload record of deploys and config changes. Ace weighs a flag change the same way it weighs a deploy, but LaunchDarkly itself never opens an incident. | | | | --- | --- | | Category | Feature flags | | Ace reads | Flag and segment changes from the audit log, flag state and targeting, evaluations, flag automation, projects and environments | | Ace can change | Flag on/off, percentage rollout, restore a flag to an earlier version, only after an admin allows actions | | Opens incidents | No. Changes join the change ledger. | | Setup | Settings form. Ace adds its own webhook when the token allows it. | ## What Ace reads While it investigates a workload with LaunchDarkly under **Investigation sources**, Ace reads: - Flag and segment changes from the audit log around the incident, with the exact before and after. - A project's flags in one environment, most recently modified first, with their evaluation status. - One flag's targeting per environment. Ace gets rule attributes and operators but never the values, and it counts individually targeted contexts without listing them. - Evaluations per variation in the incident window, compared with the window before. These come from LaunchDarkly's usage API, which is in beta. - Flag automation: scheduled changes, approvals, triggers, workflows, and guarded or progressive rollouts. For the System Map, Ace reads projects with their environments and flags, inside your allowlists. On LaunchDarkly **Enterprise** it also reads code references, so it knows which repositories use each flag. ## What Ace can change Ace proposes LaunchDarkly actions only after an admin turns on **Allow Ace to run approved actions** for the connection. Each one needs approval by default, and earned autonomy can let it run under an Automatic policy for a workload and environment you confirm. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). | Action | What it does | Approval | Undo | | --- | --- | --- | --- | | Flag kill switch | Turns one flag off (or on) in one environment. It can reverse a flag change that Ace tied to the incident. | Approval, or Automatic under your policy | Yes. Restores the previous state as a new action that is approved separately. | | Percent rollout | Rolls a variation out by percentage in stages. You set the wait between stages (60 to 3,600 seconds). | Approval, or Automatic under your policy | Yes. Serves the previous default again. | | Flag version restore | Restores a flag's targeting to how it was before a recorded change. | Approval, or Automatic under your policy | Yes | Ace applies these safeguards: - Every change carries a comment that names the Ace action, so you can find it in the LaunchDarkly audit log. - Every change checks the flag version Ace saw. LaunchDarkly refuses the write if the flag changed since. - Ace refuses to toggle a flag whose changes need a LaunchDarkly approval in that environment. - Ace rechecks the incident between stages. Settings shows which actions the token's role allows. Writer, Admin and Owner allow them, and Reader and No access don't. For custom or inline roles, Settings shows that the permission is checked when the action runs. ## What opens an incident Nothing. LaunchDarkly changes appear on your workloads' timelines and feed investigations, but they never open or join an incident as an alert. Ace records flag and segment changes in the projects and environments you allow. It keeps the exact fields that changed and any on/off or percentage transition, and it ignores other audit entries. Changes Ace made itself are recorded as part of that action. Each change belongs to: 1. The workload that routes this connection, if one does. Select **Changes from LaunchDarkly** under the workload's **Integration alerts** sources. 2. Otherwise, each workload mapped to a repository that uses the flag, from code references. This needs LaunchDarkly Enterprise. 3. Otherwise, no workload. Changes arrive by webhook as they happen. LaunchDarkly retries a delivery only once and doesn't guarantee it, so Ace also polls the audit log every 2 minutes, whether or not a workload routes the connection. ## Before you start - The **admin** role in Ace. - A token creator with at least the role you give Ace, for example Writer. A personal access token can't exceed its creator's role. - LaunchDarkly **Enterprise**, if you want Ace to match changes to workloads through code references. ## Credentials and permissions | Credential | Where to create it | Role or actions | Why Ace needs it | | --- | --- | --- | --- | | Access token (needed) | **Organization settings** > **Authorization** > **Tokens** | Reader role | Read flags, projects, environments and the audit log. | | Same token (only for the kill switch) | Same | `updateOn` on `proj/*:env/*:flag/*`, or the Writer role | Turn a flag on or off. | | Same token (only for percent rollouts) | Same | `updateFallthrough`, or the Writer role | Roll a variation out in stages. | | Same token (only for version restores) | Same | `updateRules`, `updateTargets`, `updateOffVariation`, `updatePrerequisites`, or the Writer role | Restore a flag to an earlier version. | | Same token (only for automatic webhook setup) | Same | `createWebhook`, `deleteWebhook`, `updateUrl`, `updateSecret`, `updateOn`, `updateStatements` on `webhook/*` | Let Ace add, maintain and remove its own webhook. | | Signing secret (only for manual webhook setup) | Shown on the webhook once **Sign this webhook** is checked | None | Verifies that deliveries come from LaunchDarkly. | ## Set up ### 1. Create an access token in LaunchDarkly 1. Open **Organization settings** > **Authorization** > **Tokens** and create a token named Ace. 2. Set the role to **Reader**. For approved changes, use **Writer** or an inline policy with the actions above. Add the webhook actions if Ace should set up its own webhook. Keep the latest API version. 3. Save the token and copy it, since LaunchDarkly shows it once. ### 2. Connect LaunchDarkly in Ace 1. Open **Settings** > **Data sources** and choose **LaunchDarkly**. 2. Fill in the form: | Field | What to enter | | --- | --- | | **Access token** | The token you created. | | **Instance** | Commercial (app.launchdarkly.com), EU (app.eu.launchdarkly.com) or Federal (app.launchdarkly.us). | | **Projects** | Optional. A comma-separated allowlist of project keys. | | **Environments** | Optional. A comma-separated allowlist of environment keys, such as `production`. | 3. Select **Connect LaunchDarkly**. Before it saves, Ace checks that it can reach your instance and read projects, flags and the audit log. ### 3. Add the changes webhook If the token has the webhook permissions, Ace creates a signed webhook named `ace-` plus a short ID, tagged `ace`. It covers every flag and segment change in every environment, and your allowlists decide what Ace records. The **Changes** section then says "Ace set up a signed webhook for flag and segment changes in every environment." Ace rechecks the webhook hourly and removes it when you disconnect. Without those permissions, the connection still works on polling. To add the webhook yourself: 1. In the **Changes** section, select **Show manual setup** if needed, and copy the webhook URL. 2. In LaunchDarkly, open **Integrations** > **Webhooks**, choose **Add integration**, enter the URL and check **Sign this webhook**. 3. Add a policy statement that allows all actions on `proj/*:env/*:flag/*` and `proj/*:env/*:segment/*`. Accept the Integration Terms and Conditions and save. 4. Copy the signing secret, paste it into Ace and select **Save**. ### 4. Optional: attach changes to a workload Open the workload in Ace, turn on **Integration alerts**, and select **Changes from LaunchDarkly**. Changes from this connection then belong to that workload. ### 5. Optional: allow actions Turn on **Allow Ace to run approved actions**. To let flag actions earn Automatic, confirm their workload and environment on the same panel. ## Confirm it's working - The connection checks the token's identity, projects, flags and audit log. A wrong instance fails with a message that there is no LaunchDarkly API there. - The **Changes** badge shows **Not set up**, **Waiting for first event**, **Verified** or **Last delivery failed**. Change a flag in an allowed environment, and the badge turns **Verified** when the first signed delivery arrives, and the change appears on the workload's timeline. ## Disconnect 1. In Ace, open the LaunchDarkly connection under **Settings** > **Data sources** and select **Disconnect**. Ace deletes the webhook it created, or shows a warning that names the webhook to remove by hand if it can't. 2. In LaunchDarkly, delete the Ace access token. If you added the webhook yourself, delete it under **Integrations** > **Webhooks**. ## Next steps - [PostHog](https://tryace.dev/docs/integrations/posthog) - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals) - [All integrations](https://tryace.dev/docs/integrations) --- Source: https://tryace.dev/docs/integrations/pagerduty # PagerDuty > Connect PagerDuty so a page joins the Ace incident already open for the same workload, and Ace reads timelines, alerts, on-call and service context during investigations. Source: https://tryace.dev/docs/integrations/pagerduty Ace reads PagerDuty to see who was paged, for what, and how the response went. It never adds notes to a page or resolves one. | | | | --- | --- | | Category | Paging | | Ace reads | Incidents, incident timelines, alerts, on-call, service context, change events, similar incidents | | Ace can change | Nothing in your incidents (read-only). A full-access key lets Ace add its own webhook subscription. | | Opens incidents | A page joins the Ace incident already open for the same workload. A high-urgency page opens one if none is open. | | Setup | Settings form | ## What Ace reads - PagerDuty incidents around the Ace incident, plus earlier ones still open, with status, urgency, priority, service and assignees. - Incident timelines: trigger, notifications, acknowledgements, escalations, reassignments and resolve. Ace also reads time to first acknowledgement and responder notes. - Who was on call for a service's escalation policy, by level. - The alerts behind an incident, with source integration, dedup key, severity and links. - Change events tied to an incident, or from shortly before the Ace incident. - Service dependencies, business services and escalation rules. - Past similar, related and outlier incidents. This needs PagerDuty AIOps, and Ace says so when your account doesn't have it. Ace doesn't use PagerDuty to verify a fix, because a page closes when a person resolves it. For the System Map, Ace discovers your PagerDuty services and their dependencies. ## What Ace can change Nothing in your PagerDuty incidents. Ace never acknowledges, resolves or annotates a page, and PagerDuty has no actions in Ace. A full-access key lets Ace create its own webhook subscription, and nothing else (see [Wire alerts](#3-wire-alerts)). ## What opens an incident A page usually joins the Ace incident already open for the same workload, which your alert sources opened. If no incident is open, a high-urgency page opens one. Events arrive through an account webhook subscription, and Ace also polls PagerDuty every 2 minutes in case a delivery goes missing. | PagerDuty event | What Ace does | | --- | --- | | `incident.triggered`, `incident.reopened` | Joins the Ace incident already open on the workload, or opens one. | | `incident.acknowledged` | Joins an open incident as evidence only. | | `incident.priority_updated` | Evidence only. Ace ignores it if the page is already resolved. | | `incident.resolved` | Marks the page resolved in Ace. | Low-urgency pages only inform investigations, and so do pages whose incident type you list in **Not reliability incidents**, for example Security or Customer request. If you list **Services**, Ace keeps only pages from those services, even though the subscription covers the whole account. Severity comes from the page's priority. P1 or SEV1 is critical, P2 or SEV2 high, P3 or SEV3 medium, and P4, P5, SEV4 or SEV5 low. A high-urgency page without a priority is high. A page reaches an incident only when a workload routes PagerDuty. In the workload's settings, under **Investigation triggers**, turn on **Integration alerts** and check PagerDuty under **Alert sources**. Routing PagerDuty alone doesn't narrow which sources Ace reads for that workload. ## Before you start - The **admin** role in Ace. See [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions). - **Admin** or **Account Owner** in PagerDuty, to create an API key. ## Credentials and permissions Ace uses one **general access REST API key**. A read-only key is enough if you add the webhook subscription yourself. | Credential | Where to create it | Access | Why Ace needs it | | --- | --- | --- | --- | | API key (needed) | **Integrations** > **Developer Tools** > **API Access Keys** | Read-only general access key | Read incidents, timelines, alerts, on-call, services and change events, and poll for new pages. | | API key (only for alerts) | Same | Full access general key | Let Ace add and remove its webhook subscription. | ## Set up ### 1. Create the API key 1. As an Admin or the Account Owner, open **Integrations** > **Developer Tools** > **API Access Keys** > **Create New API Key**. 2. Set **Description** to Ace and tick **Read-only API Key**. Leave it unticked only if Ace should add its webhook subscription. 3. Select **Create Key** and copy the key, because PagerDuty shows it only once. Then select **Close**. ### 2. Connect PagerDuty in Ace 1. Open **Settings** > **Data sources** and choose **PagerDuty**. 2. Fill in the form: - **API key**: the key you copied. - **Service region**: **US (api.pagerduty.com)** or **EU (api.eu.pagerduty.com)**. - **Services** (optional): service IDs to keep, comma-separated, for example `PF9KMXH`. - **Not reliability incidents** (optional): incident types that inform investigations but never open an Ace incident, comma-separated. Low-urgency pages never open one either. 3. Select **Connect PagerDuty**. Ace saves only if the key passes validation. ### 3. Wire alerts If the key has full access, Ace adds an account-scoped webhook subscription for incident events, named `ace-`, and stores the signing secret PagerDuty issues. Events from every service reach Ace, including services you add later. Ace rechecks the subscription hourly and removes it when you disconnect, and the **Alerts** section reports what it set up. If the key is read-only, the **Alerts** section says Ace can't add the subscription, though polling still brings pages in. To add the subscription yourself: 1. Copy the webhook URL Ace shows. 2. In PagerDuty, open **Integrations** > **Generic Webhooks (v3)**, add a subscription at the account scope, and paste the URL. 3. Select `incident.triggered`, `incident.acknowledged`, `incident.reopened`, `incident.resolved` and `incident.priority_updated`, then select **Add Webhook**. 4. Copy the signing secret. PagerDuty shows it once, when it creates the subscription. 5. In Ace, paste it into the signing secret field of the **Alerts** section and select **Save**. ## Confirm it's working - The connection shows **Connected** once Ace checks the key against your region, reads your services (or each one you listed), and reads incidents. - The **Alerts** section shows **Waiting for first event** until the first signed delivery turns it **Verified**. Ace polls in the meantime. ## Disconnect 1. In Ace, open **Settings** > **Data sources** > **PagerDuty** and select **Disconnect**. Ace deletes its webhook subscription, and if it can't, it tells you so you can remove `ace-` in PagerDuty yourself. 2. If you added the subscription yourself, delete it in **Integrations** > **Generic Webhooks (v3)**. 3. In PagerDuty, delete the API key. ## Next steps - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works) - [incident.io](https://tryace.dev/docs/integrations/incident-io) - [All integrations](https://tryace.dev/docs/integrations) --- Source: https://tryace.dev/docs/integrations/incident-io # incident.io > Connect incident.io so a declared incident joins the Ace incident already open for the same workload, and Ace reads updates, alerts, on-call and your service catalog during investigations. Source: https://tryace.dev/docs/integrations/incident-io Ace reads incident.io to see what your team declared, when impact started, which services were affected and who responded. Your service catalog, with owners and dependencies, appears on the System Map. Ace never writes to incident.io. | | | | --- | --- | | Category | Paging | | Ace reads | Declared incidents, updates and timeline, attached alerts, escalations, follow-ups, service catalog, on-call | | Ace can change | Nothing (read-only) | | Opens incidents | A declared incident joins the Ace incident already open for the same workload. It opens one if none is open. | | Setup | Settings form | ## What Ace reads - Incidents declared around the Ace incident, plus earlier ones still open or updated. Ace reads status, severity, type, lead, impact timestamps and custom fields. - An incident's updates, timeline items and activity log. - Attached alerts and their sources, linked incidents, escalations and follow-ups. - Catalog services that match a name, ID or alias, with their owners. - Who was on call in a schedule during the window. Ace skips test and tutorial incidents. It doesn't use incident.io to verify a fix, because an incident closes when a person moves it on. For the System Map, Ace discovers the services in your catalog with their owner teams, dependencies and tier, leaving out archived entries. ## What Ace can change Nothing. Ace doesn't declare, update or close incidents, post updates, change your catalog or set anything up in incident.io. ## What opens an incident A declared incident usually joins the Ace incident already open for the same workload, which your alert sources opened. If no incident is open, a live declared incident opens one. Events arrive through a webhook endpoint you add, and Ace also polls incident.io every 2 minutes in case a delivery goes missing. Ace accepts **Public incident created (v2)**, **updated (v2)** and **status updated (v2)**, and ignores other events. | incident.io status category | What Ace does | | --- | --- | | Triage, Live, Paused | Joins the Ace incident already open on the workload, or opens one. | | Learning, Closed, Declined, Canceled, Merged | Marks the incident.io incident resolved in Ace. | These only inform investigations: - Test and tutorial incidents. - Retrospective incidents, either by mode or because their impact ended before they were declared. - Incidents whose type you list in **Not reliability incidents**, for example Security or Customer support. Severity comes from the severity's name, for example Critical, Major, Minor, or SEV and P levels. If the name doesn't match, Ace uses the severity's rank. A declared incident reaches an Ace incident only when a workload routes incident.io. In the workload's settings, under **Investigation triggers**, turn on **Integration alerts** and check incident.io under **Alert sources**. Routing incident.io alone doesn't narrow which sources Ace reads for that workload. ## Before you start - The **admin** role in Ace. See [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions). - Permission to create API keys and webhook endpoints in incident.io. ## Credentials and permissions | Credential | Where to create it | Permissions | Why Ace needs it | | --- | --- | --- | --- | | API key (needed) | **Settings** > **API keys** | View data | Read incidents, updates and attached alerts, and poll for new incidents. | | Same key (recommended) | Same | View catalog | Read your service catalog and show services on the System Map. | | Same key (recommended) | Same | View on-call resources | Read on-call schedules. | | Webhook signing secret (only for alerts) | **Settings** > **Webhooks**, on the endpoint's page | None | Verifies that deliveries come from incident.io. | Without View catalog or View on-call resources, Ace still connects, but those reads report missing access and the System Map shows no incident.io services. ## Set up ### 1. Create the API key 1. Open **Settings** > **API keys** (`https://app.incident.io/settings/api-keys`) and choose **Add new**. Name it Ace. 2. Select **View data**, **View catalog** and **View on-call resources**. 3. Create the key and copy it, since incident.io shows it once. ### 2. Connect incident.io in Ace 1. Open **Settings** > **Data sources** and choose **incident.io**. 2. Fill in the form: - **API key**: the key you copied. - **Service catalog type** (optional): the catalog type that lists your services, by name or ID. When it's empty, Ace looks for one named Service. - **Not reliability incidents** (optional): incident types that inform investigations but never open an Ace incident, comma-separated. 3. Select **Connect incident.io**. Ace saves only if the key passes validation. ### 3. Add the webhook 1. Copy the webhook URL from the **Alerts** section. 2. In incident.io, open **Settings** > **Webhooks** and add an endpoint with the URL. 3. Subscribe to **Public incident created (v2)**, **updated (v2)** and **status updated (v2)**, then save. 4. Copy the signing secret from the endpoint's page. 5. In Ace, paste it into the signing secret field of the **Alerts** section and select **Save**. Ace rejects deliveries that have a mismatched signature or are too old. ## Confirm it's working - The connection shows **Connected** with your key's name once Ace reads its identity and lists incidents. - The **Alerts** section shows **Waiting for first event** until the first signed delivery turns it **Verified**. Ace polls in the meantime. ## Disconnect 1. In Ace, open **Settings** > **Data sources** > **incident.io** and select **Disconnect**. 2. In incident.io, delete the webhook endpoint under **Settings** > **Webhooks**. Ace didn't create it, so Ace doesn't remove it. 3. Delete the API key under **Settings** > **API keys**. ## Next steps - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works) - [PagerDuty](https://tryace.dev/docs/integrations/pagerduty) - [All integrations](https://tryace.dev/docs/integrations) --- Source: https://tryace.dev/docs/mcp # Ace MCP server > Give coding agents, code reviewers and other MCP clients read-only access to Ace's incident context, lessons from past fixes, your operating rules and the current state of a workload. Source: https://tryace.dev/docs/mcp The Ace MCP server gives any [Model Context Protocol](https://modelcontextprotocol.io) client read access to what Ace knows about your incidents, so the agent that changes your code can see what happened in production, what was tried before and how your team wants the work done. All tools are read-only, and the server can't change anything in Ace or in your systems. Ace changes live systems only through its own actions, under your [approval policy](https://tryace.dev/docs/concepts/actions-and-approvals). | | | | --- | --- | | Server URL | `https://app.tryace.dev/api/mcp` | | Transport | Streamable HTTP (stateless, JSON responses) | | Authentication | Bearer token in the `Authorization` header | | Protocol versions | `2025-06-18`, `2025-03-26`, `2024-11-05` | | Tools | 5, all read-only | Common uses: - A coding agent loads an incident's root cause, evidence and proposed actions before it opens a pull request. - A reviewer checks whether a change repeats a fix that made a similar incident worse. - Agents read your [operating rules](https://tryace.dev/docs/concepts/operating-rules) before they diagnose or change anything. - You check a workload's open incidents, early warnings and recent changes before a deploy. ## Before you start - Creating a token needs the admin role, because a token reads your whole organization. See [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions). - Your MCP client must support remote servers over Streamable HTTP with a custom `Authorization` header. For stdio-only clients, use the `mcp-remote` bridge under **Other clients** in [Step 2](#step-2-connect-your-client). - Ace authenticates with access tokens, so skip any OAuth sign-in your client offers for this server. ## Step 1: Create a token 1. In Ace, open **Settings** > **MCP**. 2. Under **Create a token**, name the token after its use, such as `Claude Code - payments repo`. 3. Choose when it expires: in 30, 90 (the default), 180 or 365 days. 4. Select **Create token**. 5. Copy the token now, because Ace shows it once and stores only a hash. Tokens start with `ace_mcp_`. Store the token as an environment variable in your shell profile or secret manager, then reference the variable in your client's configuration so the token stays out of config files: ```bash ``` > **A token can read your whole organization** > > A token reads every incident, workload and operating rule in your Ace organization, whoever created it, so treat it as a production read credential. Keep it out of source control, issue one per client or person, and use the shortest expiry that works. ## Step 2: Connect your client **Claude Code** Create `.mcp.json` in the repository root. Claude Code expands `${ACE_MCP_TOKEN}` from your environment when it connects, so the file holds no secret and can be committed. ```json title=".mcp.json" { "mcpServers": { "ace": { "type": "http", "url": "https://app.tryace.dev/api/mcp", "headers": { "Authorization": "Bearer ${ACE_MCP_TOKEN}" } } } } ``` Or use the CLI. The single quotes save the variable reference instead of its value, so the token itself never lands in the config: ```bash claude mcp add --transport http ace https://app.tryace.dev/api/mcp --header 'Authorization: Bearer ${ACE_MCP_TOKEN}' ``` Run `/mcp` to check that `ace` is connected with five tools. On first use, Claude Code asks you to approve the project's servers. **Cursor** Add Ace to `.cursor/mcp.json` in your project, or to `~/.cursor/mcp.json` for every project. Cursor reads `${env:ACE_MCP_TOKEN}` from the environment it started in, so launch it from a terminal where the variable is set. ```json title=".cursor/mcp.json" { "mcpServers": { "ace": { "url": "https://app.tryace.dev/api/mcp", "headers": { "Authorization": "Bearer ${env:ACE_MCP_TOKEN}" } } } } ``` Check **Cursor Settings** > **MCP** for `ace` with five tools. **VS Code** Add Ace to `.vscode/mcp.json` in your workspace, where VS Code uses `servers` as the top-level key. This configuration asks for the token on first start and stores it securely, so the file holds no secret: ```json title=".vscode/mcp.json" { "inputs": [ { "type": "promptString", "id": "ace-mcp-token", "description": "Ace MCP token", "password": true } ], "servers": { "ace": { "type": "http", "url": "https://app.tryace.dev/api/mcp", "headers": { "Authorization": "Bearer ${input:ace-mcp-token}" } } } } ``` To read the token from your environment instead, use `"Authorization": "Bearer ${env:ACE_MCP_TOKEN}"` and drop the `inputs` block. In the Chat view, switch to agent mode and select the tools icon to check that Ace's five tools are listed. **Codex** Codex reads the token from the environment variable on each connection: ```bash codex mcp add ace --url https://app.tryace.dev/api/mcp --bearer-token-env-var ACE_MCP_TOKEN ``` Or edit `~/.codex/config.toml`: ```toml title="~/.codex/config.toml" [mcp_servers.ace] url = "https://app.tryace.dev/api/mcp" bearer_token_env_var = "ACE_MCP_TOKEN" ``` Run `codex mcp list` to check it. **Other clients** Point any remote-capable client at `https://app.tryace.dev/api/mcp` over Streamable HTTP, with the header `Authorization: Bearer `. The older SSE transport isn't offered. For stdio-only clients such as Claude Desktop, use the open-source [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) bridge, which needs Node.js 18 or later. Put the whole header value in an environment variable and leave no space after `Authorization:`, because some clients split arguments on spaces. ```json title="claude_desktop_config.json" { "mcpServers": { "ace": { "command": "npx", "args": [ "-y", "mcp-remote", "https://app.tryace.dev/api/mcp", "--transport", "http-only", "--header", "Authorization:${ACE_AUTH_HEADER}" ], "env": { "ACE_AUTH_HEADER": "Bearer ace_mcp_..." } } } } ``` To test from a terminal, list the tools with `curl`: ```bash curl -s https://app.tryace.dev/api/mcp \ -H "Authorization: Bearer $ACE_MCP_TOKEN" \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' ``` ## Tools | Tool | Parameters | What it does | | --- | --- | --- | | `get_incident_context` | incident (string, required): Incident ID or Ace incident reference such as INS-1018-ab12. | The complete Ace incident context: what happened, root cause with supporting and contradicting evidence, open questions, evidence timeline, topology, and every proposed or running action with its constraints and pull request. Use it before reviewing or changing code for an incident. | | `search_incidents` | query (string, optional); status ("open" or "resolved" or "any", optional); limit (integer, 1–25, optional) | Find Ace incidents by meaning (similar causes and fixes across the organization) and by text in the headline, reference or workload name. Returns references to pass to get_incident_context. | | `get_lessons` | incident (string, optional): Incident ID or Ace reference such as INS-1018-ab12.; query (string, optional): The problem in a sentence, to find similar incidents.; workload (string, optional): Workload name or ID.; limit (integer, 1–10, optional) | What Ace learned from verified outcomes: fixes that worked on a workload, fixes to avoid (they made an incident worse, or repeatedly did not recover), and similar past incidents with the actions taken and how each was judged. Give an incident, a description of the problem, or a workload. Use it before proposing a fix. | | `get_operating_rules` | applies_to ("all" or "diagnosis" or "remediation" or "code_change" or "live_operation", optional): Only the rules for this kind of work (rules for everything are always included). | The workspace's operating rules: how its administrators want diagnoses, recommendations, pull requests and live operations done. Follow them. They narrow what to do and never grant permissions. | | `get_workload_context` | workload (string, required): Workload name or ID. | A workload right now: its open incidents, its open early warnings (forecasts and capacity risks) and its changes (deploys, configuration, infrastructure) in the last 24 hours. | Pass an incident as its Ace reference, such as `INS-1018-ab12`, or its ID, and a workload as its name or ID. Lookups only see your organization. ### get_incident_context Returns the full incident as Markdown and as structured data, including: - headline, workload, severity, status and a link to the incident - what happened and what triggered it - the root cause with its confidence, supporting and contradicting evidence, and open questions - the evidence timeline, with source links where available - matched services and the dependency path, when known - notes from Ace and your engineers - every proposed or running action, with its progress, constraints and pull request link ```json title="Arguments" { "incident": "INS-1018-ab12" } ``` ### search_incidents Finds incidents by meaning and by text in the headline, reference or workload name. Matches by meaning come first with a similarity score, followed by the most recent text matches with severity and status. Each result includes the reference to pass to `get_incident_context`. Without `query`, it lists the most recent incidents. `status` defaults to `any` and `limit` defaults to 10. ```json title="Arguments" { "query": "orders stuck after a schema migration", "status": "resolved", "limit": 5 } ``` ### get_lessons Returns what Ace learned from verified outcomes. Pass at least one of `incident`, `query` or `workload`, and `limit` defaults to 5. The result has three parts: - **Fixes that worked** lists fixes that recovered the workload and never made it worse, with counts. - **Fixes to avoid** lists fixes that made an incident worse, or that failed to recover the workload at least twice and never recovered it. - Similar past incidents, with the actions Ace took in each and how each was judged. Fix history needs a workload, because Ace tracks fixes per workload. It takes the workload from `workload`, or from the workload of `incident`. See [How memory works](https://tryace.dev/docs/concepts/memory). ```json title="Arguments" { "workload": "checkout-api", "query": "p99 latency spike after deploy" } ``` ### get_operating_rules Returns your workspace's [operating rules](https://tryace.dev/docs/concepts/operating-rules) and the kinds of work each one covers. `applies_to` filters to one kind of work, and rules that apply to everything are always included. ```json title="Arguments" { "applies_to": "code_change" } ``` ### get_workload_context Returns a workload's open incidents and open early warnings, where each early warning is a forecast or a capacity risk with when the limit is expected to be hit. The result also lists up to 20 changes from the last 24 hours, such as deploys and configuration or infrastructure changes. ```json title="Arguments" { "workload": "checkout-api" } ``` ## Example prompts Each prompt lists the tools it should call, though most clients pick tools on their own once Ace is connected. **Fix an incident with full context** ```text Load Ace incident INS-1018-ab12 and our operating rules for pull requests. Then fix the root cause in this repository. Don't repeat any fix that Ace lists under "Fixes to avoid" for this incident's workload. ``` Tools: `get_incident_context`, `get_operating_rules` with `applies_to: "code_change"`, `get_lessons` with `incident`. **Review a pull request against past incidents** ```text Review this diff. Use Ace to find past incidents on checkout-api that touched the same code path, and flag anything this change repeats from a fix that made an incident worse or didn't recover it. ``` Tools: `search_incidents`, `get_lessons` with `workload`. **Check a workload before you deploy** ```text Before I deploy payments-worker, check Ace for open incidents, early warnings and changes in the last day. Tell me if anything argues for waiting. ``` Tools: `get_workload_context`. **Write a post-incident summary** ```text Summarize Ace incident INS-1018-ab12 for our incident review: what happened, the root cause with its evidence, what was changed, and what's still unknown. ``` Tools: `get_incident_context`. ## Security and access - All five tools are marked read-only in their MCP annotations. - A token belongs to the organization and reads all of its incidents, workloads and rules, whoever created it. Only admins can create, see and revoke tokens. - Ace stores only a hash of each token, and a token expires after the period chosen when it was created, at most 365 days. - Requests with a browser `Origin` header other than Ace's own get `403`, so call the server from agents, IDEs and CLIs. - Ace logs each tool call with the token, the tool, the incident read and the result, and keeps the log for 90 days. **Settings** > **MCP** shows when each token was last used. - A tool result is capped at 200,000 characters. Longer incident context is cut off and ends with a link to the incident in Ace. > **Treat incident content as untrusted** > > Incident context includes alert text, log lines, error messages, Slack replies and third-party pages, any of which can contain prompt injection. Keep tool-call approval on in your client, review what the agent does with the results, and don't let incident content alone justify a code or infrastructure change. ### Revoke a token 1. In Ace, open **Settings** > **MCP**. 2. Under **Tokens**, select **Revoke** next to the token. The next request with that token gets `401`. Revoked and expired tokens stay in the list with their status, so you can see which clients need a new one. ## Limits and errors | Limit | Value | | --- | --- | | Calls per token | 120 per minute | | Calls per organization, across all tokens | 600 per minute | | Requests per client address | 300 per minute | | Failed authentications per client address | 20 per minute | | Request body | 64 KB | | Tool result | 200,000 characters | Each request carries one JSON-RPC message, and batches aren't supported. | Status | Meaning | What to do | | --- | --- | --- | | `401` | The token is missing, malformed, revoked or expired. The body is `{"error": "Invalid Ace MCP token."}`. | Check the `Authorization: Bearer ace_mcp_...` header, and check that the environment variable is set where your client runs. If the token was revoked or has expired, create a new one. | | `403` | The request came from a web page (`Origin` header). | Call the server from an agent, IDE or CLI. | | `413` | The request body is larger than 64 KB. | Tool arguments are short strings, so this usually means a malformed request. | | `429` | A rate limit was reached. The `retry-after` header gives the seconds to wait. | Wait that long before you retry. If you hit the organization limit, spread the work over time or run fewer agents at once. | Tool errors, such as an unknown incident or a missing argument, come back as a normal tool result with `isError: true` and a message, such as `No incident INS-0000-xxxx in this organization.` An unknown tool returns JSON-RPC error `-32602`, and an unknown method returns `-32601`. ## Next steps - [How memory works](https://tryace.dev/docs/concepts/memory): What get_lessons and search_incidents draw on. - [Operating rules](https://tryace.dev/docs/concepts/operating-rules): Write the rules your agents read with get_operating_rules. - [Security and data](https://tryace.dev/docs/security/security-and-data): What Ace reads, keeps and protects. --- Source: https://tryace.dev/docs/security/security-and-data # Security and data > How Ace accesses your systems, what data it reads and keeps, how credentials and personal data are protected, and how your data stays inside your organization. Source: https://tryace.dev/docs/security/security-and-data Ace reads from the tools you connect and changes nothing unless an admin allows it. This page covers what Ace can access, what it stores, how that data is protected and how to delete it. For SOC 2 Type II reports and subprocessors, see the [Trust Center](https://trust.vratix.com). ## Read-only by default Connecting a source gives Ace read access. Write access is a separate step an admin takes for each integration, and it can be turned off again. | Integration | Read access | Write access | | --- | --- | --- | | AWS | A read role in your account that never changes anything and is never given write access. It explicitly denies reading data inside your resources: S3 objects, DynamoDB items, SQS messages, Secrets Manager secrets, SSM parameters and full SQL text from Performance Insights. | A separate, optional actions role whose operations come in groups, each off until you choose it. | | Kubernetes | A read-only reader Deployment that you run in your cluster. It connects out to Ace, so Ace needs no inbound access to your cluster API. | A separate, optional writer Deployment for approved actions, so read access never implies write access. | | GitHub | The Ace GitHub App, with read access to the repositories you choose. | Ace opens pull requests and runs your deploy workflow only for approved actions. | | Other integrations | The credential you provide. | Only after an admin turns on actions for that integration, and only for operations the credential allows. Ace never proposes an operation it knows the credential can't perform. | With actions allowed, every change to code or infrastructure needs approval of its exact plan. The exception is earned autonomy, where an admin made an action automatic on a workload where it has a verified record of fixing incidents without making anything worse. Updates that only change what your team sees in a tool, such as an annotation or an issue's state, run without approval and can be undone from the incident. An admin can also stop all actions at once. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). Until our team activates your workspace, Ace runs no investigations and no actions. ## What Ace reads - When it investigates a workload, Ace reads only the sources selected for that workload, plus any source that routes alerts to it. From GitHub it reads only the repositories linked to the workload. Admins choose these in the workload's settings. - In Slack, Ace receives messages in channels it has been added to and in direct messages to it. - For metric watches, Ace reads metrics from your providers. CloudWatch watches stay off until an admin turns them on, because those reads are billed to your AWS account. - The AWS actions role can technically see some values that hold secrets, such as Lambda environment variables, but Ace doesn't store them. - Ace keeps Kubernetes Secret values out of evidence. ## What Ace keeps Ace stores what it needs to investigate and to show you what happened: - incidents, the evidence collected and investigation results, where evidence is made of snippets from logs, metrics, traces and changes rather than full copies of your data - alert events and HTTP check results - action plans, approvals and the result of each run - a change ledger of deploys and other changes - your System Map, built from your AWS accounts - [memory](https://tryace.dev/docs/concepts/memory), meaning incident summaries and learned patterns - notes and Slack replies from your team, after [redaction](#redaction) Your workspace has a history window, which sets how far back you can view incidents and activity in Ace. Older activity is kept but shown as outside the window. To change the window, [talk to our team](https://tryace.dev/book-demo). Some short-lived records are deleted on a schedule. Metric samples for metric watches go after 28 days, and MCP access records, which show which token read which incident, go after 90 days. ## Credentials Ace encrypts integration credentials, Slack installation tokens, webhook secrets and Kubernetes connector credentials at rest. [MCP tokens](https://tryace.dev/docs/mcp) are stored only as hashes and shown once, when created. Ace stores no AWS credential because it assumes the role you created, using an external ID unique to your organization. Disconnecting an integration deletes the stored connection and credentials in Ace, but it doesn't revoke the credential at the provider or remove what you created on your side, such as a CloudFormation stack, a Helm release, the GitHub App installation or the Slack app. Remove those at the provider. ## Tenant isolation Ace resolves every request to the caller's organization and limits every read and write to it. Memory, including similar-incident search, covers only your organization, and nothing Ace learns in one organization is used in another. MCP tokens are scoped to one organization, and each person belongs to one organization. ## Redaction Ace removes secrets and personal data from text before storing it. From Slack replies and incident notes, Ace removes credentials, tokens, keys, signed URLs, email addresses, payment card numbers, phone numbers and IBANs. It also removes other personal or secret details it detects, such as customer names, addresses, account numbers and health details. Each removed value becomes a marker naming its type, such as `[REDACTED_EMAIL]`, before the text reaches memory, investigations or the [MCP server](https://tryace.dev/docs/mcp). From log text read from your providers, Ace scrubs credentials and common personal data, including email addresses and IPv4 addresses, before the text is used in an investigation or stored as evidence. Ace also redacts HTTP check responses and removes sensitive fields from AWS responses. For Kubernetes, it keeps only the shape of configured values. Redaction targets known kinds of secrets and personal data and can miss some, so avoid pasting secrets into Slack or notes. ## AI models Ace uses models from a third-party AI provider to investigate incidents, interpret Slack messages, detect personal data for redaction and run similar-incident search. For subprocessors, see the [Trust Center](https://trust.vratix.com). Ace treats text from your systems and your team as data, never as instructions, and before activation it sends nothing to the AI provider. ## Audit logs | What | What's recorded | | --- | --- | | Actions | An append-only record of every plan decision with who made it and the exact plan, every approval, every run, and every policy change with its before and after values. You can see an action's activity on the incident. | | Incident attempts | An append-only record of every investigation attempt and every question Ace asked, with who answered. | | MCP access | Which token read which incident, kept for 90 days. | ## Deleting data - When an admin deletes a workload, Ace first stops its checks and watches, stops running investigations and voids action plans awaiting approval, so nothing acts on a workload that's going away. It then deletes the workload's incidents, HTTP check history, memory, change events and deployment links, and finally the workload itself. - Disconnecting an integration deletes its stored connection and credentials in Ace. - Deleting a metric watch deletes its history, but incidents it opened are kept. - You can delete your own notes, and nobody who reads the incident later sees a deleted note. ## Hosting Ace is a hosted service at `app.tryace.dev`, and the Kubernetes connector runs inside your cluster and connects out to Ace. ## Next steps - [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions): Who can connect integrations, change workloads and approve actions. - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): What Ace can change and how changes are approved. - [How memory works](https://tryace.dev/docs/concepts/memory): What Ace remembers and how it stays inside your organization. --- Source: https://tryace.dev/docs/security/roles-and-permissions # Roles and permissions > The two workspace roles in Ace, what admins and members can each do, how approval permissions work, and how people sign in and join. Source: https://tryace.dev/docs/security/roles-and-permissions Ace has two workspace roles. Admins control what Ace can see and do, and members follow investigations, work incidents and approve the actions an admin lets them approve. ## Roles **Settings** > **Team** describes the roles like this: | Role | Description in Ace | | --- | --- | | **Admin** | Changes settings, invites people and approves any action. | | **Member** | Follows investigations and approves what you allow below. | The person who creates a workspace becomes its first admin. A workspace always keeps at least one admin, so the last admin can't be demoted or removed. ## What each role can do | | Admin | Member | | --- | --- | --- | | **Workspace** | | | | View incidents, workloads and the System Map | Yes | Yes | | Change workspace settings: name, timezone, how much Ace posts, operating rules, pausing all watches | Yes | No | | Complete guided setup | Yes | No | | **Team** | | | | Invite people, revoke invites, change roles, remove members | Yes | No | | Link teammates' Slack accounts to their Ace accounts | Yes | No | | **Integrations** | | | | Connect, edit, validate and disconnect integrations | Yes | No | | Allow actions on an integration, including the AWS actions role | Yes | No | | Choose which Slack channels Ace posts to | Yes | No | | Create and revoke [MCP tokens](https://tryace.dev/docs/mcp) | Yes | No | | **Workloads** | | | | Create and rename a workload | Yes | Yes | | Add HTTP checks | Yes | Yes | | Mute or tune watched signals | Yes | Yes | | Link GitHub repositories to a workload | Yes | No | | Choose which alerts start a workload's investigations | Yes | No | | Choose which sources Ace reads for a workload | Yes | No | | Add or delete custom watches | Yes | No | | Confirm which AWS resources a workload's actions may change | Yes | No | | Delete a workload, which deletes its data | Yes | No | | **Incidents** | | | | Start an investigation, in Ace or from Slack | Yes | Yes | | Investigate again, add notes, close or resolve an incident | Yes | Yes | | **Actions** | | | | View action policies and approval permissions | Yes | Yes | | Approve actions that aren't production changes | Yes | Yes | | Approve production changes | Yes | With an approval permission | | Undo an update Ace made to an observability tool | Yes | Yes | | Set action policies (**Off**, **Approval**, **Automatic**) and stop all actions | Yes | No | | Grant approval permissions | Yes | No | | Manage deployment mappings | Yes | No | Acting from Slack, including starting investigations and approving, also needs the person's Slack account linked to their Ace account, so Ace can check their role and permissions. The person who installed Slack is linked automatically, and an admin links everyone else in **Settings** > **Team**. ## Approval permissions In every environment, changes Ace makes through a provider count as production changes, except observability updates such as an annotation or a muted alarm. Fix pull requests and changes you report yourself count as production changes when they're in production. Any admin can approve a production change, and so can a member with an approval permission that covers it. Any member can approve everything else. An admin grants approval permissions in **Settings** > **Team**, or attaches them to an invite so the person can approve as soon as they join. A permission covers one or more actions, or every action, in one or more environments, or every environment. Admins don't need approval permissions because they can already approve everything. Each member's scope is shown next to their name, for example "Approves every production action" or "Can't approve production actions". Someone without permission who tries to approve a production change is told it needs an admin or a member with approval permission. Approval permissions decide only who may approve, while your action policy decides whether an action can run without approval. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals). ## MCP tokens Only admins can create and revoke tokens for the [Ace MCP server](https://tryace.dev/docs/mcp), because a token is scoped to your organization and gives read-only access to all of its incidents, workloads and operating rules, whoever created it. Tokens last 90 days by default and 365 days at most. Ace stores only a hash of each token and records which token read which incident. ## Sign-in People sign in with **Continue with Google** or **Continue with GitHub**, and one Ace account can link both. Sessions last 7 days and renew while you use Ace. Ace is invite-only, so a new account can be created only for an email address with a pending invite or one our team has given access to. ## Invites 1. An admin opens **Settings** > **Team**, or the **Team** step of guided setup. 2. The admin enters email addresses and chooses **Member** (the default) or **Admin**. 3. The invitee signs in with Google or GitHub, using the invited email address, and chooses to join the workspace or create a separate one. Invites are valid for 7 days, and an admin can revoke one before it's used. Each person belongs to one workspace. ## Next steps - [Security and data](https://tryace.dev/docs/security/security-and-data): What Ace reads, what it keeps and how it's protected. - [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): Action policies, approvals and earned autonomy. - [Ace MCP server](https://tryace.dev/docs/mcp): Create a token and connect your coding agent. - [Quickstart](https://tryace.dev/docs/quickstart): Set up a workspace and run your first investigation.