# Ace product: a team of agents for your cloud operations > Ace's agents investigate incidents, prepare fixes for review, verify recovery and keep what they learn, working from one map of your infrastructure. Start with incidents. Ace is a team of cloud agents built by Vratix, Inc. Its agents work from one graph of your infrastructure, the telemetry your services emit and the code that runs on them. ## Map the system before investigating it Ace builds a graph of your connected resources and their dependencies. Its agents use the same map to find affected callers, shared databases and services that could fail next. - Resource and service relationships - Observed dependencies and potential blast radius - Missing source coverage made visible ## Investigate in parallel, share the findings Infrastructure, telemetry and code agents pursue different leads at the same time and write to one shared incident record. A database finding redirects the trace search. A trace points to the change that needs review. When new evidence arrives, the investigation picks it up instead of starting again. - Logs, metrics, traces and recent changes - One incident for an alert storm, not hundreds - Hypotheses checked against contradictory evidence - Open questions kept alongside the diagnosis ## Prepare the fix for review Ace prepares fix and revert pull requests with the incident context attached. A fix carries the patch and, when the failure reproduces, a regression test, and is reviewed for correctness, data safety and operations before it opens. If Ace's own pull request fails your checks, it makes one follow-up fix. Rollbacks, Kubernetes changes and connector actions such as a Vercel rollback go through the same plan and approval. Your policies govern execution and your engineers merge. - Exact target, diff and expected effect - Approval requirements set by your team - Proposals and execution results kept together ## Check that service has recovered Ace revisits the original symptoms and checks affected dependencies. If recovery fails or the incident returns, it can open a follow-up investigation with the previous action and evidence attached. Retry budgets keep that loop bounded. - Recovery checks tied to the diagnosis - Remaining symptoms and regressions surfaced - Incomplete outcomes kept open for follow-up ## Keep the lesson, prevent the repeat Findings, engineer feedback and action outcomes become system memory. Ace searches past incidents across your organization by meaning, not just by name, to recognize familiar failures and avoid repeating unsuccessful work. It prepares lasting fixes and regression checks for the cause behind the incident. - Similar past incidents found across your organization - Working fixes and failed approaches remembered - Known causes carried into the next investigation ## Let results earn more autonomy Begin with read-only investigations and reviewed proposals. Production changes need an approver your administrator names. Delegate supported actions by target, with approval rules and execution limits. Ace keeps the diagnosis, authorized change and observed result together, so your team can decide what to delegate next. - Read-only investigation first - Named approvers for production changes - A record of decisions, changes and outcomes ## Work where your team already works Each agent posts in the incident's Slack thread as its own teammate. Approve a plan with a message, ask about it, and follow each step as it runs. Coding agents can pull the full incident context through Ace's MCP server, and scheduled scans flag wasted spend in connected sources. - Slack updates, questions and approvals in one thread - Read-only MCP server for incident context and search - Cost findings from Trigger.dev and LiteLLM ## How an investigation starts - An Alertmanager alert or an Amazon SNS notification, such as a CloudWatch alarm, delivered to Ace's [alert ingest API](https://app.tryace.dev/docs/api/public). - An alert or webhook from a connected source: Sentry issue alerts, Grafana alerts, PostHog alerts and failed Trigger.dev runs. - An Ace HTTP health check that fails against a URL configured on a workload. - A request from the Ace app or Slack. Signals from every source join the affected workload's open incident, so an alert storm becomes one investigation instead of hundreds. The incident closes only when all of its signals are quiet. ## What Ace can change today | Action | What happens | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Fix pull request | Ace prepares a minimal code or infrastructure-as-code repair in an isolated sandbox, adds a regression test when the failure is reproducible, reviews it for correctness, data safety and operations, and opens a pull request with the incident context. | | Revert pull request | An exact reversal of a change, opened as a pull request. | | Rollback | Redeploys a known-healthy revision through a GitHub Actions workflow an administrator has confirmed. | | Kubernetes change | Restart, scale, patch configuration or roll back to a captured revision in namespaces you grant to a separate writer identity. | | Vercel rollback or promotion | Instant rollback to a previous production deployment, or promotion of a deployment. | | Trigger.dev replay | Replays failed runs in stages, for example 10 first and then the rest, each run at most once. | | PostHog flag rollout | Rolls a feature flag out in stages, for example 10%, 50% and 100%, with signal checks between stages. | | PlanetScale change | Schema changes through deploy requests (and their revert), and bounded backfills. These always need a person's approval and never run automatically. | If Ace's own pull request fails your required checks, it makes one follow-up fix without weakening tests or checks. Your engineers merge pull requests and your pipeline deploys them, unless you turn on automatic merge for a service where Ace has earned it. Ace does not run `terraform apply` or change live infrastructure outside these actions. Every connector action needs an administrator's grant first. ## How autonomy is earned New organizations start in shadow mode: Ace records what it would do without doing it. Teams can then require approval for each plan, by its exact content, in the app or Slack. Automatic execution is a separate policy for one confirmed workload and destination. It needs at least five verified recoveries and no worsened outcomes in that scope, stays within a daily cap, and announces each run in Slack with a cancellation countdown before it writes anything. Production changes need an approver: an organization administrator, or a member an administrator has granted that action type and environment. When someone else tries to approve in Slack, Ace mentions the approvers in the thread. Automatic merge is off by default. Where it is on, Ace merges only its own earned-autonomy pull requests, after required checks and every Ace reviewer pass, and after a cancellation countdown in Slack. ## Kubernetes An outbound connector for EKS, GKE, AKS and on-premises clusters, installed with a pinned Helm chart (or a manifest) from the open-source [ace-kubernetes-connector](https://github.com/vratix-dev/ace-kubernetes-connector) repository. It supports Kubernetes 1.29 to 1.37 and works with Argo CD or Flux. Your cluster API is never exposed. Each cluster has its own identity and namespace scope, and Secret values never enter evidence. Agents read resources, events, logs, rollouts, services, network policies and node conditions. Changes such as restarting, scaling or patching a workload use a separate identity you grant, and go through the same plan, approval and audit flow as every other action. Automatic repair is narrower still: expiring, reversible namespace patches to values you approve in advance. ## Example investigation (illustrative) Orders stalled after a schema change. A reporting transaction held a relation lock, a migration's DDL queued behind it, and application reads and outbox delivery queued behind the DDL. Ace's agents connected `pg_locks`, a request trace and the deployment that retried lock acquisition without a timeout. The proposed recovery paused retries and cancelled only the waiting DDL, with approval required. The prevention pull request added a lock timeout, bounded retries and a contention regression test. Recovery checks confirmed latency and outbox delivery returned to normal. This example is synthetic; see the [worked write-up](https://tryace.dev/resources/evidence-backed-root-cause-analysis). ## Related - [Investigations](https://tryace.dev/investigations) - [Integrations](https://tryace.dev/integrations) - [Security](https://tryace.dev/security) - [Pricing](https://tryace.dev/pricing)