An automated fix should be easier to review than a command pasted into an incident channel. The reviewer needs the exact target, the change, the evidence behind it and the conditions that will stop execution.
Here is a practical review format for teams introducing automated production changes. The examples describe engineering controls to evaluate. They are not a promise that every action type is supported by Ace or your deployment.
Bind approval to a specific change
“Roll back the orders service” leaves too much open. A useful proposal identifies the environment, deployment, current revision and intended replacement. For a pull request, it identifies the repository, base commit and patch the reviewer saw.
Use a record like this:
Incident: ACE-842
Reason: deployment 842 introduced unbounded migration retries
Target: orders-platform / production / migration runner
Expected current revision: <immutable revision>
Proposed change: <patch or configured workflow + inputs>
Evidence: <source references and observation times>
Checks before execution: <target and system preconditions>
Approval: <reviewer, plan version, expiration>
Stop if: <changed revision, missing evidence, failed check>
Recovery checks: <symptoms, scope, observation window>
Fallback: <reviewed response if the action fails>
The placeholders must be resolved before execution. A service label is useful for people but is not a substitute for the provider’s resource identity.
Approval should become invalid when a material part of the plan changes. If another deployment lands while the reviewer is reading, refresh the proposal. Do the same when the patch, workflow inputs or target scope changes. Quietly carrying an old approval onto a new plan defeats the review.
Separate the permissions for each step
Reading a repository, opening a pull request, merging it and deploying it are distinct capabilities. Grant only the capabilities the workflow needs.
For GitHub, configure branch protection or rulesets for required reviews and checks, and inspect bypass permissions. Required reviews are useful only if the automation cannot quietly bypass them. GitHub also provides settings for dismissing stale approvals and requiring approval of the most recent reviewable push. Review the protected-branch controls against your actual workflow.
For deployment workflows, bind the action to a configured target and an allowed workflow. Reject an ambiguous service-to-deployment mapping. A system that cannot establish the target should leave a proposal for an engineer to resolve.
Decide what recovery means before running the change
Take an orders incident with a backed-up outbox. A healthy API response does not establish that pending events are being delivered. The review needs both checks.
Write the observation plan in terms the on-call engineer can verify:
- Original symptom: request error rate and latency for the affected endpoint and workload.
- Shared dependency: the lock or connection wait that supported the diagnosis.
- Downstream progress: outbox age and delivery rate, including whether the backlog is draining.
- New damage: errors in dependent services and failures introduced by the change.
- Observation window: enough traffic and time to judge the result, chosen for this system.
Avoid a universal five-minute “healthy” rule. A low-traffic service or a periodic worker may need a different window. If the required telemetry is absent, record recovery as unverified.
Handle the response you never received
Suppose a provider accepts an action, but the connection drops before the client receives the response. Retrying immediately can create a second change. Declaring failure may also be wrong: the first action could still be running.
Keep an operation identifier and reconcile against provider state before deciding to retry. Use the provider’s idempotency mechanism where available. When execution cannot be established, show an unknown outcome and a next check. The AWS Builders’ Library on idempotent APIs explains the underlying retry problem.
The action history should distinguish these states: proposed, approved, dispatched, accepted, completed and recovery verified. A rejected request, a failed workflow and an unknown result need their own records. Do not compress them into one green or red badge.
Exercise the boundaries in a sandbox
Before granting production execution, run cases that test the controls themselves:
- Change the target revision after approval. Execution should stop or require renewed review.
- Request a resource outside the allowed scope. It should remain untouched.
- Reject or expire an approval. No provider operation should be dispatched.
- Lose the provider response. Confirm that reconciliation does not duplicate the action.
- Let the action succeed while the symptom persists. Recovery must remain unresolved.
- Make the fallback unsafe because the system has changed. It should require a fresh decision rather than an automatic reversal.
Retain the provider-side result as well as the application’s action record. That is how another engineer can distinguish “the UI says blocked” from “nothing reached production.”
Leave a fix that survives the incident
Immediate recovery and prevention deserve separate review. Cancelling a queued migration may restore requests, while bounding retries and testing lock contention address why the same queue could return.
Link the follow-up patch, regression and owner to the incident. Later, verify that the patch shipped and the test exercises the failure. Memory is useful when it changes the next decision, not when it merely makes an old incident easier to search.
Ace starts with read-only investigations and separately authorized actions. Review the access model and use these checks to decide which work your team is ready to delegate.