# Proactive detection

> How Ace watches your workloads on its own with HTTP checks, metric watches and early warnings, how it learns what normal looks like, and how to tune it.

Source: https://tryace.dev/docs/concepts/proactive-detection

Besides your own alerts, Ace raises three kinds of alerts itself. Ace HTTP checks call your endpoints, metric watches read your providers' metrics, and early warnings fire when a series is heading toward a known limit.

Watches cover latency, errors, throughput, saturation and cost. Ace learns each series' normal range and opens an incident when a deviation persists. Most of this needs no setup, because Ace builds the watches from your System Map and connected sources.

## Ace HTTP checks

An Ace HTTP check calls one of your endpoints on a schedule and measures latency and errors. Add one when you create a workload, with **HTTP health check** under **Which alerts start an investigation?**, or later in the workload's settings under **Investigation triggers**. Members and admins can both add HTTP checks.

| Setting | What you set |
| --- | --- |
| **Endpoint** | **Base URL**, **Path**, **Method** and **Authentication** (**None**, **API Key Header** or **Bearer Token**). Credentials are stored securely. |
| **Request payload** | The request body, for methods that send one. |
| **Frequency & limits** | How often Ace checks, from the frequencies available to your workspace: **Light** (4 per hour), **Standard** (12 per hour), **Intensive** (30 per hour) or **Aggressive** (60 per hour). To stay within your API's limits, turn on **Rate limits** and set **Max checks per hour** and, if you want, **Max checks per day**. |
| **Anomaly detection** | **Adaptive baseline** adjusts thresholds to the traffic Ace observes. Turn it off to set a **P95 latency threshold (ms)** and an **Error rate threshold (%)** yourself. |

Ace confirms a failed check with follow-up checks before it raises anything, so a single blip doesn't open an incident. A failure that holds opens an **Error Rate Spike**, **Latency Regression** or **Chronic Slowness** incident on the workload.

Ace investigates high and critical HTTP check incidents on its own, and for error rate spikes warning severity is enough. Failures that clear on their own teach Ace that the check is noisy. See [How memory works](https://tryace.dev/docs/concepts/memory).

## Metric watches

A metric watch is one series that Ace reads from a provider on a schedule and compares with its learned normal range.

| What it measures | Read |
| --- | --- |
| Latency, error rate, throughput, failures | Every 5 minutes |
| Saturation and queue age | Every 15 minutes |
| Cost | Daily |

### Which series Ace watches

Ace creates watches for three sets of resources:

- the resources a workload names exactly on the System Map, from the resources, names, tags and repositories you chose when you created it
- what those resources contain
- resources one runtime dependency away, such as the workload's database, cache or queue

Ace never matches resources by a loose name match, which would put watches on unrelated resources, and it creates watches only on connections the workload's investigations read. When the System Map changes, Ace updates the list and keeps your tuning and mutes, and a watch whose resource leaves the workload stops running.

The tables below list every watch Ace can create, where "Limit" is the known ceiling that early warnings measure against.

#### AWS (CloudWatch)

| Resource | Watches | Limit |
| --- | --- | --- |
| RDS instance | CPU, Database connections, Freeable memory (a drop is bad), Free storage (a drop is bad), Read latency, Write latency | CPU 100%, connections at the instance's maximum (when known), free storage 0 |
| Aurora cluster | CPU, Database connections, Freeable memory (a drop is bad), Serverless capacity (ACU, when the maximum capacity is known) | CPU 100%, connections at the maximum, ACU at the maximum capacity |
| RDS Proxy | Client connections, Database connections, Connection borrow latency | None |
| ElastiCache cluster | Cache hit rate (a drop is bad), Engine CPU, Memory used, Connections, Evictions | CPU and memory 100%, connections at `maxclients`, or 65,000 by default |
| Lambda function | Duration p95, Errors, Invocations (a drop is bad), Throttles, Concurrent executions (when reserved concurrency is set) | Concurrency at the reserved concurrency |
| ECS service | CPU, Memory | 100% |
| SQS queue | Age of oldest message, Messages visible | Message age at the queue's retention period (4 days by default) |
| Application Load Balancer | Target response time p95, Target 5xx responses, Requests (a drop is bad) | None |
| DynamoDB table | Read throttle events, Write throttle events, System errors | None |
| EC2 instance (not in an Auto Scaling group) | CPU, Status check failed | CPU 100% |
| Auto Scaling group | In-service instances, when the group's maximum size is known | The group's maximum size |
| Kinesis stream | Iterator age, Write throughput exceeded | Iterator age at the stream's retention period (24 hours by default) |
| API Gateway REST API | Latency p95, 5xx errors | None |
| CloudFront distribution | 5xx error rate | None |
| Each AWS service above | The account's daily spend on that service | None |

Ace watches an Auto Scaling group as a whole, without separate watches on its instances. Daily spend comes from AWS billing metrics, which exist only when billing alerts are turned on in the account.

> **CloudWatch watches are off until an admin turns them on**
>
> CloudWatch metric reads are billed to your AWS account, so CloudWatch watches are listed but not read until an admin turns on **Watch CloudWatch metrics**. The switch is under **CloudWatch watches** in **Settings** > **General** and in the AWS settings, and a notice in a workload's **Watched signals** links to it too.
>
> Watches on other providers run either way. See [AWS](https://tryace.dev/docs/integrations/aws).

#### Kubernetes (through Prometheus)

Ace watches Deployments, StatefulSets and DaemonSets through your organization's Prometheus source, which is a Grafana Prometheus data source or Amazon Managed Prometheus. The organization must have exactly one Prometheus source, so Ace knows which one to query.

| Watch | Read | Limit |
| --- | --- | --- |
| CPU of limit | Every 15 minutes | 100% |
| Memory of limit | Every 15 minutes | 100% |
| Container restarts | Every 5 minutes | None |
| Unavailable replicas | Every 5 minutes | None |

#### Other integrations

| Integration and resource | Watches | Limit |
| --- | --- | --- |
| Supabase project | Postgres connections, CPU load, Memory used, Disk used | Connections at the project's maximum, and 100% for CPU, memory and disk |
| New Relic service | Latency p95, Error rate, Throughput (a drop is bad) | None |
| New Relic host | CPU, Memory used, Disk used | 100% |
| Honeycomb dataset | Latency p95, Error rate | None |
| Cloudflare zone or proxied DNS record | Edge 5xx share | None |
| Trigger.dev task | Failed runs | None |
| Inngest function | Failed runs | None |
| LiteLLM gateway | Spend, Failure rate, Latency p95 | None |
| LiteLLM model | Failure rate, Latency p95 | None |
| PostHog project | Pageviews (a drop is bad), Exceptions | None |
| Datadog APM service | Latency and errors for each of the service's trace metrics on the System Map | None |

### Your own watches

You can add a watch for any other series, such as a business metric:

1. In the workload's **Watched signals**, choose the source and connection.
2. Write the query, name the watch and pick what it measures.
3. If a drop is the problem, turn on **A drop is bad (hit rate, traffic, free storage)**.
4. Select **Preview** to read the last hour and check the query.
5. Select **Add watch**.

| Source | What you enter |
| --- | --- |
| Amazon CloudWatch | **Namespace**, **Metric**, dimensions, **Statistic** and **Region** |
| Datadog | **Metric query** |
| New Relic | **Account ID** and an **NRQL aggregate** |
| Honeycomb | **Dataset**, **Calculation** and column |
| Grafana | **Prometheus datasource UID** and **PromQL** |
| Amazon Managed Prometheus | **PromQL** |
| Supabase | **Project ref** and **Metric** (connections, CPU, memory or disk) |
| Cloudflare | **Host** |
| Trigger.dev | **Task** |
| Inngest | **Function** |
| LiteLLM | **Metric** (spend, requests, failure rate or latency p95) and, optionally, **Model** |
| PostHog | **Event** |

A workload can have up to 50 of your own watches, and only admins can add or delete them.

## How Ace decides a series is abnormal

Ace learns each watch's normal range from the series' own history, including daily and weekly patterns, so a nightly batch job or a Monday-morning peak isn't flagged.

A new watch shows **Learning** until Ace has enough history to judge it. Where the provider keeps history, Ace reads it first, so many watches can alert right away, though old deviations in that history never open anything. **Start alerting now** ends learning early.

One odd sample doesn't fire a watch, because the deviation has to hold. The watch clears once the series is back to normal, and Ace ignores changes too small to matter, such as a tiny move on a very steady series.

A step change fires. If the series stays at the new level, Ace eventually learns it as normal and the incident closes without counting as a recovery. Slow drift moves the normal range with it, so a forecast against a known limit is what catches it.

A sample past a threshold you set always counts as deviating, even while the watch is learning.

| Severity | When |
| --- | --- |
| High | Past your threshold, or far from normal. Ace investigates on its own. |
| Warning | Any other firing deviation. Recorded on the incident and watched. |

Like any alert, a firing watch opens an incident on the workload or joins the open one. See [How Ace works](https://tryace.dev/docs/concepts/how-ace-works).

## Early warnings

Any watch with a known limit in the tables above can raise an early warning before anything fails.

| | Forecast | Capacity risk |
| --- | --- | --- |
| Raised when | The series' trend is on course to reach its limit soon | The series is running close to its limit |
| Severity | Higher the sooner the limit will be reached | Higher the closer the series is to its limit |
| Clears | When the trend no longer projects a breach | When the series moves back away from its limit |
| In Ace | **Forecast** with "Reaches its limit" and the time | **Capacity risk** with "At N% of its limit" |

When a forecast comes true, it becomes a capacity risk and keeps its investigation.

At warning or above, an early warning starts an investigation and is posted once to your alert channels, while **Info** forecasts stay in Ace. If the workload already has an open incident, Ace adds the early warning to it as evidence.

Ace never acts on its own on an early warning, because nothing has failed yet. Any preventive step waits for a person, whatever your [action policy](https://tryace.dev/docs/concepts/actions-and-approvals) says.

A workload's open early warnings are listed in its **Watched signals**, and the [Ace MCP server](https://tryace.dev/docs/mcp) returns them from `get_workload_context`.

## Tuning detection

A workload's **Watched signals** settings list each watch with what it measures, how often it's read, and its current value against its normal range and its limit or threshold. Each watch has one of these statuses:

| Status | Meaning |
| --- | --- |
| **Normal** | Within its learned normal. |
| **Learning** | Building its baseline. |
| **Waiting for data** | No sample read yet. |
| **Deviating** | Firing now. |
| **Can't read** | The last reads failed. The error is shown. |
| **Muted** | Turned off. |

**Checked less often** means your workspace hit its read limit for that provider, so Ace reads the watch less often until that clears.

Open a watch to change it:

| Control | What it does |
| --- | --- |
| **Sensitivity** | **Low**, **Normal** (the default) or **High**. Low needs a bigger deviation to fire, and High catches smaller ones. |
| **Alert above** / **Alert below** | A fixed threshold, with **Save threshold**. A sample past it always counts as deviating, even while the watch is still learning. |
| **Start alerting now** | Ends learning early for this watch. **Start alerting for all** does it for every learning watch on the workload. |
| Mute | Stops the watch. If the watch is part of an open incident, muting ends its alert, and the incident closes if nothing else in it is still firing. |
| **Delete** | For your own watches, admins only. "Ace stops watching this series and deletes its history. Incidents it opened are kept." |

Members and admins can tune and mute watches. Only admins can add or delete custom watches.

To stop every watch, an admin can turn on **Pause all metric watches** in **Settings** > **General**. Ace then stops reading your providers for every workload's watches and closes what they raised, and the watches restart when you resume.

Closing a watch's incident as **Expected behavior** or **Not actionable** also tunes it, and after enough of these closes Ace stops raising that alert on that workload. See [How memory works](https://tryace.dev/docs/concepts/memory).

## Next steps

  - [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): What happens after a check, watch or early warning opens an incident.
  - [How memory works](https://tryace.dev/docs/concepts/memory): How repeated closes quiet a noisy signal, and how that fades.
  - [Integrations](https://tryace.dev/docs/integrations): Connect the sources Ace builds watches from.