Proactive detection

Besides your own alerts, Ace raises three kinds of alerts itself. Ace HTTP checks call your endpoints, metric watches read your providers' metrics, and early warnings fire when a series is heading toward a known limit.

Watches cover latency, errors, throughput, saturation and cost. Ace learns each series' normal range and opens an incident when a deviation persists. Most of this needs no setup, because Ace builds the watches from your System Map and connected sources.

Ace HTTP checksCopy Anchor Link

An Ace HTTP check calls one of your endpoints on a schedule and measures latency and errors. Add one when you create a workload, with HTTP health check under Which alerts start an investigation?, or later in the workload's settings under Investigation triggers. Members and admins can both add HTTP checks.

Ace confirms a failed check with follow-up checks before it raises anything, so a single blip doesn't open an incident. A failure that holds opens an Error Rate Spike, Latency Regression or Chronic Slowness incident on the workload.

Ace investigates high and critical HTTP check incidents on its own, and for error rate spikes warning severity is enough. Failures that clear on their own teach Ace that the check is noisy. See How memory works.

Metric watchesCopy Anchor Link

A metric watch is one series that Ace reads from a provider on a schedule and compares with its learned normal range.

Which series Ace watchesCopy Anchor Link

Ace creates watches for three sets of resources:

• the resources a workload names exactly on the System Map, from the resources, names, tags and repositories you chose when you created it
• what those resources contain
• resources one runtime dependency away, such as the workload's database, cache or queue

Ace never matches resources by a loose name match, which would put watches on unrelated resources, and it creates watches only on connections the workload's investigations read. When the System Map changes, Ace updates the list and keeps your tuning and mutes, and a watch whose resource leaves the workload stops running.

The tables below list every watch Ace can create, where "Limit" is the known ceiling that early warnings measure against.

AWS (CloudWatch)Copy Anchor Link

Ace watches an Auto Scaling group as a whole, without separate watches on its instances. Daily spend comes from AWS billing metrics, which exist only when billing alerts are turned on in the account.

Kubernetes (through Prometheus)Copy Anchor Link

Ace watches Deployments, StatefulSets and DaemonSets through your organization's Prometheus source, which is a Grafana Prometheus data source or Amazon Managed Prometheus. The organization must have exactly one Prometheus source, so Ace knows which one to query.

Other integrationsCopy Anchor Link

Your own watchesCopy Anchor Link

You can add a watch for any other series, such as a business metric:

1. In the workload's Watched signals, choose the source and connection.
2. Write the query, name the watch and pick what it measures.
3. If a drop is the problem, turn on A drop is bad (hit rate, traffic, free storage).
4. Select Preview to read the last hour and check the query.
5. Select Add watch.

A workload can have up to 50 of your own watches, and only admins can add or delete them.

How Ace decides a series is abnormalCopy Anchor Link

Ace learns each watch's normal range from the series' own history, including daily and weekly patterns, so a nightly batch job or a Monday-morning peak isn't flagged.

A new watch shows Learning until Ace has enough history to judge it. Where the provider keeps history, Ace reads it first, so many watches can alert right away, though old deviations in that history never open anything. Start alerting now ends learning early.

One odd sample doesn't fire a watch, because the deviation has to hold. The watch clears once the series is back to normal, and Ace ignores changes too small to matter, such as a tiny move on a very steady series.

A step change fires. If the series stays at the new level, Ace eventually learns it as normal and the incident closes without counting as a recovery. Slow drift moves the normal range with it, so a forecast against a known limit is what catches it.

A sample past a threshold you set always counts as deviating, even while the watch is learning.

Like any alert, a firing watch opens an incident on the workload or joins the open one. See How Ace works.

Early warningsCopy Anchor Link

Any watch with a known limit in the tables above can raise an early warning before anything fails.

When a forecast comes true, it becomes a capacity risk and keeps its investigation.

At warning or above, an early warning starts an investigation and is posted once to your alert channels, while Info forecasts stay in Ace. If the workload already has an open incident, Ace adds the early warning to it as evidence.

Ace never acts on its own on an early warning, because nothing has failed yet. Any preventive step waits for a person, whatever your action policy says.

A workload's open early warnings are listed in its Watched signals, and the Ace MCP server returns them from get_workload_context.

Tuning detectionCopy Anchor Link

A workload's Watched signals settings list each watch with what it measures, how often it's read, and its current value against its normal range and its limit or threshold. Each watch has one of these statuses:

Checked less often means your workspace hit its read limit for that provider, so Ace reads the watch less often until that clears.

Open a watch to change it:

Members and admins can tune and mute watches. Only admins can add or delete custom watches.

To stop every watch, an admin can turn on Pause all metric watches in Settings > General. Ace then stops reading your providers for every workload's watches and closes what they raised, and the watches restart when you resume.

Closing a watch's incident as Expected behavior or Not actionable also tunes it, and after enough of these closes Ace stops raising that alert on that workload. See How memory works.

Next stepsCopy Anchor Link