AceDocs

Actions and approvals

Every change Ace can make, how your action policy decides what needs a person, who can approve, how undo works, how Ace verifies recovery, and how earned autonomy lets an action run automatically.

Ace can roll back a bad deploy, scale a service, flip a feature flag or open a pull request. Each change is an action with an exact plan, and your action policy decides whether a person approves the plan or Ace runs it on its own.

Every action needs approval by default, and after any change Ace checks that the workload actually recovered.

How an action works

  1. Ace prepares each plan from live state, naming the exact change, the resource it touches, a preflight check, how to undo it (when it can) and the alert that will show it worked.
  2. Your policy decides whether the plan waits for approval or runs automatically.
  3. A person approves the exact plan on the incident page under Remediation, or in Slack. If you choose Edit values and change a value, such as the replica count, Ace prepares the plan again and it needs a new approval, because an approval covers only the exact plan.
  4. Ace applies the change and confirms it took effect.
  5. Ace verifies recovery on the incident's own alerts, then keeps watching in case the problem comes back.

Where actions come from

  • Undoing a correlated change. When a deploy, a flag or config change, or an infrastructure change lines up with the incident, Ace proposes undoing it. If a person made that change during or shortly after an incident on the workload, Ace always asks before undoing it, whatever the policy.
  • Operations on running systems, such as scaling a service or failing over a load balancer, for causes outside code. Ace doesn't propose these on a low-confidence diagnosis, since a live change aimed at the wrong cause can make things worse.
  • Fix pull requests for code defects, for your team to review.
  • Work after the fix. Once the workload recovers, Ace can propose redoing lost work, such as redriving an SQS dead-letter queue or replaying failed runs, and these always wait for the fix to deploy.
  • Requests in Slack, such as @Ace scale checkout to 6 tasks. Ace picks the matching operation for the workload, prepares it from live state and posts it for approval, and @Ace help <workload> lists what you can ask for there.

Three kinds of actions

KindWhat it doesHow it runs
RemediationChanges the workload to end the incident.Under your action policy. Judged on recovery.
Observability updateChanges only what people see about the incident, such as annotations, silences, downtimes, mutes and issue state.Automatically by default once allowed on the connection, unless a policy for that exact action says otherwise. Must be undoable. Never judged on recovery.
After the fixRedoes work the incident lost.Waits for the fix to deploy and always needs approval. Never holds the incident open.

Before Ace can act

Ace's access is read-only until an admin allows changes, and each integration has its own switch:

IntegrationHow an admin allows actions
GitHubThe Ace GitHub App opens revert and fix pull requests. Roll back deploy and Redeploy run through a deploy workflow that an admin confirms for each workload under Rollback workflow in Settings > Actions > Workloads.
KubernetesThe Ace connector in your cluster makes changes, only in the namespaces confirmed for the workload. See Kubernetes.
AWSAn optional actions role with the operation groups you choose. Each group is off until you choose it, and actions run only on the resources an admin confirms for each workload. See AWS.
Other connected toolsTurn on Allow Ace to run approved actions on the connection. For tools whose only writes are observability updates, the switch is Allow Ace to update this tool during incidents. Ace never proposes an action that the connection's credentials are known to lack permission for.

PagerDuty and incident.io have no actions.

Action policy

An admin sets the policy in Settings > Actions > Workloads by choosing a workload, an action and a mode.

ModeIn AceWhat happens
OffOff, "Ace never runs this action."Ace never runs it, but can still recommend the fix.
ApprovalApproval, "A person approves each plan."Every plan waits for approval.
Approval, tracking eligibilityApproval with Track eligibility on, "Record whether each proposal would have qualified for Automatic."Every plan waits for approval, and Ace records whether each one would have qualified for Automatic. This is the default when no policy is set.
AutomaticAutomatic, "Ace acts and posts an Undo option in Slack."Ace runs qualifying plans on its own after a countdown in Slack. Locked until the action qualifies. See Earned autonomy.

Which policy applies

  • Stop all actions in Settings > Actions sets the whole organization to Off, while Ace still investigates and recommends fixes. Resume actions turns actions back on.
  • If any policy that matches the action is Off, the action doesn't run.
  • Otherwise the most specific policy wins. A policy for the exact action beats one for all actions, then a policy for the workload beats one for all workloads, and a policy for the environment beats one for all environments.
  • Change freezes, in Settings > Actions > Change freezes, are periods such as a launch when nothing runs automatically, though Ace still recommends fixes for approval. You can schedule up to 20.

Approvals

Who can approve

PlanWho can approve it
A change to productionAn admin, or a member an admin granted approval for that action (or all actions) in that environment (or all environments).
Any other planAny member.

In any environment, Ace treats these as production changes:

  • every change it makes through Kubernetes, AWS or your other connected tools, except observability updates
  • every rollback and redeploy
  • every undo of a correlated change

Fix pull requests and changes you report with I applied it count as production when the environment is production.

Admins can approve every action in every environment. To choose what a member can approve, use Settings > Team or set it when you invite them.

Two approvers

Each plan needs one approver by default. To require Two people, add an approval rule in Settings > Actions > Approvals. A rule can cover all actions or one, all workloads or one, and all environments or one, and the most specific rule applies.

The two approvals must come from different people, and the second approver sees Approve as second. Actions with two approvers never run automatically, because the rule asks for two people.

Approving on the web and in Slack

Approve on the incident page under Remediation or from the decision card Ace posts in the incident's Slack thread, since both approve the same plan. The Slack decision card has these buttons:

  • Approve, labeled with the action, such as Approve rollback or Create revert PR
  • Edit values
  • Decline
  • Ace's alternatives, as Instead: …
  • Something else…, to tell Ace what you want in your own words

Acting from Slack needs your Slack user linked to your Ace account, because Ace checks your permission when you click. If nobody answers, Ace reminds the thread, and on high and critical incidents it also mentions people who can approve, admins first.

When a fix spans several repositories, its pull requests are approved together in one decision.

Earned autonomy

Ace earns the right to act on its own one action, workload and environment at a time. Automatic stays locked for an action on a workload until the action has a verified track record there, and your policy, preflight checks and daily limits still apply after it unlocks. The workload's action settings show its Track record, such as "3 of 5 verified recoveries, 0 worsened", and why Automatic is locked.

To turn on Automatic

  • The policy names one action, one workload and one environment, and the workload's deployment mapping for that environment is confirmed. Automatic can't be set for All actions or All workloads, because a track record only means something for one action on one workload.
  • The action has at least the number of verified recoveries set in Recoveries required (5 by default, 1 to 100), with no outcome that made things worse, all in that exact workload, environment and deployment mapping.
  • Only recoveries after a deploy that the provider recorded count. A deploy you confirmed with I deployed it, or a change you reported with I applied it, never counts, since Ace can't verify those itself.
  • For Kubernetes change (an exact patch), a repair grant replaces the track record. An admin approves the exact fields Ace may change in the confirmed namespaces, with an expiry, under Settings > Data sources > Kubernetes. Only reversible changes to approved values qualify, and creates, deletes, cluster-wide changes, credential fields and list edits always need a person.

If a later outcome makes things worse, the track record stops qualifying and the action goes back to needing approval.

When an automatic action runs

A plan runs on its own only if Automatic is on and all of these hold. If any one fails, Ace posts the plan for approval with the reason:

  • The incident isn't an early warning.
  • The action doesn't write to a data store.
  • No approval rule asks for two approvers.
  • It's inside the policy's set hours (Limit to set hours) and outside any change freeze.
  • The incident isn't holding its fix for a linked incident.
  • The preflight is safe. A blocked preflight never runs, and one with cautions runs only if Run despite preflight cautions is on.
  • The action doesn't always need a person. See the catalog.
  • For Kubernetes, no GitOps controller (Argo CD or Flux) manages the object, since the controller would revert the change. The operation also isn't one that always needs a person, such as an image from a registry the workload doesn't already use.
  • For everything else, Ace tied the incident to a specific deployment, unless Run without a linked deployment is on.
  • The plan can be undone.
  • The action is under its Daily limit (5 by default, up to 100, counted per UTC day).

AWS plans always carry a preflight caution, so AWS actions run automatically only with Run despite preflight cautions on.

Before an automatic action runs, Ace posts a short countdown with an Undo option in the incident's Slack thread. If the countdown can't be posted, the action doesn't run, because nobody would have the chance to stop it.

For Fix pull request and Revert pull request, you can turn on Auto-merge pull requests, which is off by default. With it on, Ace merges when every Ace reviewer passes and checks are green, after announcing the merge in Slack so anyone can stop it.

Undo and rollback

Undo exists only where Ace can restore the exact prior state, and the catalog shows which actions have it.

  • Before an automatic action runs, Undo in the Slack countdown cancels it.
  • After an action is applied, undo is a new action that Ace prepares from the prior state, and it needs approval like any other.
  • Observability updates are undone with one click on the incident. The click is your approval.

When a change makes things worse, Ace can propose undoing it but never undoes it automatically.

Roll back deploy redeploys a known-healthy revision through the deploy workflow an admin confirmed for the workload, without reversing database changes or provisioning infrastructure. A rollback can include the matching revert pull request in the same approval, so your main branch matches what's running.

Recovery verification

Ace first confirms the action took effect, then checks the alerts that opened the incident, such as the original HTTP check, the same alert from your tool, or the metric's normal range. Passing CI, a merged pull request or someone closing the incident never counts, because none of them show the workload is healthy. See How Ace works.

ResultIn AceWhat it meansCounts toward Automatic
Verified recoveredRecoveredThe incident's alerts recovered after the change took effect.Yes, when the provider recorded the deploy
Did not recoverDid not recoverThe change took effect, but the alerts didn't recover.No
Made it worseMade it worseThe alerts got worse after the change.Blocks Automatic for that action on that scope
Recovered on its ownRecovered on its ownThe workload recovered before the change took effect, or after some other change.No

After recovery, Ace keeps watching in case the problem returns. Results feed memory, so a fix that made an incident worse on a workload is never proposed again there, and a fix that didn't recover it is proposed again only with new evidence.

Action catalog

Each operation below is a separate action with its own policy and track record, named as Ace shows it.

  • Always a person means it never runs automatically, whatever the policy. "Yes (data store)" marks a write to a data store.
  • Undo means Ace can restore the exact prior state.
  • Values you can edit before approving, such as a replica count, show their allowed range.

GitHub and deploys

ActionWhat it doesAlways a personUndo
Fix pull requestOpens a pull request with a fix for a code, config or infrastructure-as-code cause.NoYes
Revert pull requestOpens a pull request that reverts the change behind the incident.NoYes
Roll back deployRedeploys a known-healthy revision through your confirmed deploy workflow.NoYes
RedeployDeploys the revision already serving again, with no code change.NoNo
Applied by a personRecords a change you made yourself (I applied it). Ace runs nothing and only verifies recovery.YesNo

Kubernetes

ActionWhat it doesAlways a personUndo
Kubernetes changeAn exact patch, create or delete on a namespaced resource. Automatic only with a repair grant.NoNo
Kubernetes rollout restartRestarts a Deployment, StatefulSet or DaemonSet's pods.NoNo
Kubernetes scaleSets replicas on a Deployment or StatefulSet (0–1,000).NoYes
Kubernetes autoscaler updateChanges an autoscaler's minimum and maximum replicas (1–1,000) or CPU target (1–1,000%).NoYes
Kubernetes rollout undoRestores a previous revision.NoYes
Kubernetes rollout pausePauses a Deployment's rollout.NoYes
Kubernetes rollout resumeResumes a paused Deployment rollout.NoYes
Kubernetes container imageSets a container's image, pinned by digest.NoYes
Kubernetes container resourcesChanges a container's CPU and memory requests and limits.NoYes
Kubernetes environment valueSets a non-secret environment variable on a container.YesNo
Kubernetes CronJob suspendSuspends a CronJob.NoYes
Kubernetes CronJob resumeResumes a suspended CronJob.YesYes
Kubernetes Job re-runRuns a Job, or a CronJob's job, again.YesNo
Kubernetes disruption budgetChanges a PodDisruptionBudget.NoYes
Kubernetes node cordonStops new pods from scheduling on a node.YesYes
Kubernetes node uncordonLets pods schedule on a node again.YesYes
Kubernetes node taintAdds a taint to a node.YesYes
Kubernetes node taint removalRemoves a taint from a node.YesYes
Kubernetes pod deleteDeletes a pod so its controller replaces it.YesNo
Kubernetes node drainCordons a node and evicts the confirmed namespaces' pods, respecting disruption budgets. Undo only uncordons.YesNo
Argo Rollouts abortAborts an Argo Rollout.NoYes
Argo Rollouts promotePromotes an Argo Rollout.YesNo
Argo Rollouts retryRetries an aborted Argo Rollout.YesYes
Argo Rollouts undoRestores an Argo Rollout's previous revision.NoYes
Argo CD syncSyncs an Argo CD application to what Git holds now.YesNo
Argo CD rollbackRolls an Argo CD application back to an earlier history entry. Needs auto-sync off.NoYes
Argo CD auto-sync disableTurns off auto-sync for an application.YesYes
Argo CD auto-sync enableTurns on auto-sync for an application.YesYes
Flux suspendSuspends a Flux Kustomization or HelmRelease.YesYes
Flux resumeResumes a Flux Kustomization or HelmRelease.YesYes
Flux reconcileAsks Flux to reconcile a Kustomization or HelmRelease now.NoNo

Any Kubernetes change to an object a GitOps controller manages always needs a person.

AWS

Each AWS action belongs to an operation group that you turn on in the actions role, and all of them run only on resources confirmed for the workload.

ActionWhat it doesAlways a personUndo
ECS service rollbackReturns an ECS service to its previous task definition.NoYes
ECS service restartForces a new deployment of an ECS service.NoNo
ECS desired task countSets a service's desired tasks (0–5,000).NoYes
ECS task stopStops one ECS task so the service replaces it.NoNo
ECS auto scaling limitsChanges a service's scaling minimum and maximum (0–5,000).NoYes
ECS task sizeChanges a task's CPU (128–16,384 units) and memory (512–122,880 MiB).YesYes
Lambda provisioned concurrency scaling limitsChanges provisioned-concurrency scaling limits (0–10,000).NoYes
Lambda alias rollbackPoints an alias back to its previous version.NoYes
Lambda alias weighted routingShifts traffic between alias versions (weight 0–1).NoYes
Lambda reserved concurrencySets reserved concurrency (0–10,000).NoYes
Lambda provisioned concurrencySets provisioned concurrency (1–10,000).NoYes
Lambda event source mapping enable/disableTurns an event source mapping on or off.NoYes
Lambda event source batch sizeChanges an event source's batch size (1–10,000).NoYes
Lambda memory and timeoutChanges a function's memory (128–10,240 MB) and timeout (1–900 s).NoYes
Auto Scaling group desired capacitySets desired instances (0–10,000).NoYes
Auto Scaling group size limitsChanges a group's minimum and maximum size (0–10,000).NoYes
Auto Scaling process suspensionSuspends Auto Scaling processes.NoYes
Auto Scaling process resumptionResumes suspended processes.NoYes
Auto Scaling instance refreshStarts an instance refresh (minimum healthy 0–100%).YesNo
Auto Scaling instance refresh cancellationCancels a running instance refresh.NoNo
Auto Scaling instance refresh rollbackRolls back a running instance refresh.NoNo
Auto Scaling instance healthMarks an instance unhealthy so the group replaces it.NoNo
Auto Scaling instance terminationTerminates one instance in a group.NoNo
Auto Scaling group instance typeChanges a group's instance type.YesYes
EC2 instance rebootReboots an instance.NoNo
EC2 instance stopStops an instance.YesYes
EC2 instance startStarts a stopped instance.YesYes
EC2 instance typeChanges an instance's type.YesYes
EKS managed node group scalingChanges a node group's minimum, maximum and desired size (0–1,000).NoYes
RDS instance rebootReboots a database instance.Yes (data store)No
Aurora cluster failoverFails an Aurora cluster over to a replica.Yes (data store)No
RDS instance classChanges a database instance's class.YesYes
RDS parameterChanges a database parameter.Yes (data store)Yes
Aurora reader addedAdds a reader to an Aurora cluster.Yes (data store)Yes
Aurora reader removedRemoves a reader from an Aurora cluster.YesYes
Aurora Serverless v2 capacityChanges minimum and maximum capacity (0–256 ACU).Yes (data store)Yes
RDS storage and autoscaling limitGrows storage and changes its autoscaling limit (20–65,536 GiB).Yes (data store)Yes
RDS Proxy connection poolChanges the proxy's connection limits (1–100%).Yes (data store)Yes
DynamoDB provisioned capacityChanges read and write capacity (1–40,000 units).Yes (data store)Yes
DynamoDB auto scaling limitsChanges a table's scaling limits (0–40,000 capacity units).Yes (data store)Yes
DynamoDB capacity modeSwitches between provisioned and on-demand capacity.YesYes
ElastiCache node rebootReboots cache nodes.Yes (data store)No
ElastiCache replica countChanges replicas per shard (0–5).Yes (data store)Yes
ElastiCache node typeChanges the node type.YesYes
ElastiCache failoverFails a replication group over.YesNo
SQS dead-letter queue redriveAfter the fix: moves messages from the dead-letter queue back to the source queue (1–500 messages per second).Yes (data store)No
SQS visibility timeoutChanges the visibility timeout (0–43,200 s).NoYes
SQS maximum receivesChanges receives before dead-lettering (1–1,000).NoYes
Kinesis shard countChanges a stream's shards (1–10,000).NoYes
Target group health checkChanges a target group's health check.NoYes
Target group deregistration delayChanges the deregistration delay (0–3,600 s).NoYes
Load balancer target removalRemoves a target from a target group.NoYes
Load balancer target registrationAdds a target to a target group.NoYes
Load balancer traffic shiftChanges a listener's weights between target groups.NoYes
Route 53 weighted recordChanges a weighted record's weight (0–255).NoYes
Route 53 failover swapSwaps primary and secondary failover records.YesYes
CloudFront invalidationInvalidates cached paths.NoNo
API Gateway stage throttlingChanges a stage's rate (0–100,000 per second) and burst (0–100,000).NoYes
API Gateway canary promotionPromotes a stage's canary. Only a person proposes it.NoNo
API Gateway canary rollbackRolls back a stage's canary.NoNo
WAF IP set additionAdds addresses to a WAF IP set.NoYes
WAF IP set removalRemoves addresses from a WAF IP set.NoYes
WAF rule action overrideOverrides a WAF rule's action.NoYes
CodeDeploy stop and roll backStops a deployment and rolls it back.NoNo
CloudFormation update cancellationCancels a stack update in progress.NoNo
CloudFormation rollback continuationContinues a stack's stuck rollback.NoNo
AppConfig deployment stopStops an AppConfig deployment.NoNo
Step Functions execution redriveAfter the fix: redrives a failed execution.NoNo
EventBridge rule disableDisables an EventBridge rule.NoYes
EventBridge rule enableEnables an EventBridge rule.NoYes
SSM Automation runbookRuns an SSM Automation document an admin allowed.YesNo
SSM Run CommandRuns an SSM command on instances.YesNo
CloudWatch alarm actions mutedObservability update: mutes a firing alarm's actions during an approved fix. Runs on its own.NoYes
CloudWatch alarm actions restoredObservability update: restores an alarm's actions.NoYes

Vercel

ActionWhat it doesAlways a personUndo
Vercel instant rollbackRolls production back to an earlier deployment.NoYes
Vercel promotionPromotes a deployment to production.NoYes
Vercel redeployRebuilds and redeploys a deployment.NoYes
Vercel deployment cancelCancels a deployment that's still building.NoNo
Vercel rolling release stage approvalMoves a rolling release to its next stage.YesNo
Vercel rolling release abortAborts a rolling release.NoNo
Vercel Attack Challenge ModeTurns on Attack Challenge Mode for 15–1,440 minutes.YesYes
Vercel firewall IP blockBlocks an IP address.YesYes
Vercel firewall IP unblockRemoves an IP block.YesYes
Vercel environment variable changeSets an environment variable to a new value you give, then redeploys.YesYes

Cloudflare

ActionWhat it doesAlways a personUndo
Cloudflare cache purgePurges cached content for a zone.NoNo
Cloudflare Worker rollbackRolls a Worker back to an earlier version.NoYes
Cloudflare load balancer failoverTurns a load balancer pool off or on to move traffic.YesYes
Cloudflare security levelChanges a zone's security level.YesYes
Cloudflare WAF rule toggleTurns a WAF rule on or off.YesYes

Feature flags

ActionWhat it doesAlways a personUndo
LaunchDarkly flag kill switchTurns a flag off or on.NoYes
LaunchDarkly percent rolloutChanges a flag's rollout percentage in stages.NoYes
LaunchDarkly flag version restoreRestores a flag to its state before a recorded change.NoYes
PostHog flag rolloutChanges a flag's rollout percentage in stages.NoYes
PostHog flag switchTurns a flag off or on.NoYes

Background jobs

ActionWhat it doesAlways a personUndo
Trigger.dev run replayAfter the fix: replays failed runs in stages (up to 1–2,000 runs).NoNo
Trigger.dev run cancelCancels runs in stages (up to 1–2,000 runs).YesNo
Trigger.dev queue pausePauses a task's queue.YesYes
Inngest run rerunAfter the fix: reruns failed runs in stages (up to 1–500 runs).NoNo
Inngest function pausePauses a function.YesYes
Inngest function resumeResumes a paused function.YesYes
Inngest run cancelCancels runs in stages (up to 1–500 runs).YesNo

Databases

ActionWhat it doesAlways a personUndo
PlanetScale schema changeDeploys a schema change through a deploy request.YesYes
PlanetScale schema revertReverts a deployed schema change.YesNo
PlanetScale backfillUpdates rows in batches (1–1,000 rows per batch, up to 100,000 rows) in stages.YesNo
Supabase pooler pool sizeChanges the default pool size (1–3,000 connections).YesYes
Supabase read-only mode liftLifts read-only mode on a project.YesNo
Supabase disk increaseGrows the project's disk (1–60,000 GB).YesNo
Supabase project restartRestarts the project.YesNo

All PlanetScale and Supabase actions write to a data store.

LiteLLM

ActionWhat it doesAlways a personUndo
LiteLLM model fallbackRoutes a failing model to a fallback.NoYes
LiteLLM budget and limitsChanges budgets and rate limits.YesYes

Observability updates

These change only what your team sees, so once you allow them on the connection, the ones that can be undone run on their own during incidents.

ActionWhat it doesUndo
Grafana annotationMarks the incident on a Grafana dashboard.Yes
Grafana annotation removalRemoves that annotation.No
Grafana alert silenceSilences a Grafana alert for 5–240 minutes.Yes
Grafana silence expiryEnds a silence early.No
Axiom annotationMarks the incident on an Axiom dataset.Yes
Axiom annotation removalRemoves that annotation.No
Sentry issue resolveResolves a Sentry issue.Yes
Sentry issue reopenReopens a Sentry issue.No
Sentry issue assignmentAssigns a Sentry issue.Yes
Datadog monitor downtimeSilences a Datadog monitor for 5–240 minutes.Yes
Datadog downtime cancellationEnds a downtime early.No
New Relic muting ruleMutes a New Relic alert for 5–240 minutes.Yes
New Relic muting rule removalRemoves a muting rule.No
Honeycomb markerMarks the incident on a Honeycomb dataset.Yes
Honeycomb marker removalRemoves that marker.No

Staged actions, such as rollouts, replays, cancels and backfills, check the incident's alert between stages. You can change Wait between stages (60–3,600 seconds) before you approve.

Next steps

On this page