# Prometheus

> Connect Amazon Managed Service for Prometheus or an in-cluster Prometheus on Amazon EKS so Ace can query metrics during investigations and open incidents from Alertmanager alerts.

Source: https://tryace.dev/docs/integrations/prometheus

Ace queries Prometheus for the affected workload's latency, error rate, throttling, memory and restarts, and with Alertmanager connected, your alerts open incidents. Ace supports two deployment types:

- **Amazon Managed Prometheus (AMP)**. Ace queries the AMP API, and alerts arrive through an Amazon SNS topic.
- **In-cluster Prometheus (EKS)**. Prometheus Operator or kube-prometheus-stack in an Amazon EKS cluster, which Ace queries through the cluster's Kubernetes API server. Alerts arrive through an Alertmanager webhook.

Other self-hosted Prometheus servers aren't supported directly, but you can remote-write their metrics to an AMP workspace and connect that.

| | |
| --- | --- |
| Category | Kubernetes metrics and alerts |
| Ace reads | PromQL queries, scrape targets, rules and metric metadata. With alerts on, Alertmanager status |
| Ace can change | Nothing (read-only) |
| Opens incidents | Yes, from Alertmanager alerts, if you turn on alert triggers |
| Setup | Guided setup in Ace that creates a role in your AWS account |

## What Ace reads

During an investigation, Ace runs PromQL range queries over the incident window for p95 latency, request rate, error rate, CPU throttling, memory pressure and usage, restart rate and OOM kills. It also reads scrape targets, rules and metric metadata.

On AMP, Ace queries your workspace through the AMP API. On EKS, it queries the Prometheus service through the Kubernetes API server's service proxy and can read resources only in the Prometheus namespace you choose.

## What Ace can change

Ace changes nothing in Prometheus. Its role can't change rules, scrape configuration or your Alertmanager definition, and on EKS it also can't deploy workloads, edit manifests or change RBAC.

## What opens an incident

Only Alertmanager alerts open incidents, and only if you turn on **Alert triggers** during setup.

- A firing alert opens an incident or joins an open one.
- A resolved alert resolves the matching incident.
- Alerts named `Watchdog` or `InfoInhibitor`, and alerts with severity `info` or `none`, never open incidents.

An alert opens an incident only for a workload that routes it. To route it, open the workload's settings, turn on **Integration alerts** under **Investigation triggers** and select **Alertmanager via AMP** or **Alertmanager via EKS Prometheus Operator**.

Ace doesn't poll Prometheus for alerts, so without Alertmanager alerts, Ace uses Prometheus only as evidence.

## Before you start

- The **admin** role in Ace.
- Permission in AWS to create CloudFormation stacks and IAM roles.
- **AMP**: an AMP workspace and, for alerts, an AMP Alertmanager definition you can edit. The alerts stack creates an SNS topic, billed at standard Amazon SNS pricing.
- **EKS**: an EKS cluster whose authentication mode allows access entries, with Prometheus running in one namespace, and, for alerts, access to your Alertmanager configuration.

Settings shows one Prometheus connection, either AMP or EKS, so to change its settings, disconnect and connect again.

### Private networks

- **AMP**: Ace queries the AMP API with the role the setup creates and alerts arrive through Amazon SNS, so nothing in your VPC needs to be reachable. For a private EKS cluster, remote-write from your in-cluster Prometheus to AMP and connect the workspace.
- **EKS**: Ace connects from its cloud to your cluster's Kubernetes API endpoint, so the cluster needs public endpoint access, and a cluster with only private endpoint access must use AMP. For alerts, Alertmanager needs outbound HTTPS to Ace.

The [Kubernetes](https://tryace.dev/docs/integrations/kubernetes) connector runs inside the cluster and connects out to Ace, so it works with a private API endpoint.

## Credentials and permissions

Ace never stores AWS access keys. The guided setup uses CloudFormation to create an IAM role, which Ace assumes with an external ID generated for your workspace, and you review the stack in the AWS console before you create it.

| Deployment type | What the setup creates | Why Ace needs it |
| --- | --- | --- |
| AMP | A role with read-only query access to your workspace only: metrics, series, labels, metric metadata and rules | Metrics queries and rules |
| AMP, alerts on | Read access to Alertmanager status, and an SNS topic that only AMP can publish to, from your workspace. An HTTPS subscription delivers to Ace | Alertmanager status and alert delivery |
| EKS | A role that can describe your cluster only, and an EKS access entry with the AWS-managed `AmazonEKSAdminViewPolicy`, scoped to the Prometheus namespace | Read-only access in that namespace, including the service proxy used to query Prometheus and Alertmanager |
| EKS, alerts on | A webhook secret generated by Ace. Alertmanager sends it as a bearer token | Authenticates alert deliveries |

## Set up

In Ace, go to **Settings** > **Data sources** > **Prometheus** and choose your deployment type.

**AMP**

1. Choose **Amazon Managed Prometheus (AMP)**.
2. Under **Capabilities**, **Metrics querying** is always on. Turn on **Alert triggers** if you want Alertmanager alerts to open incidents.
3. Under **Configuration**, choose the **Region** and enter the **AMP Workspace ID** (`ws-...`).
4. Choose **Launch stack in AWS**, then review the stack in the AWS console and create it.
5. When the stack completes, copy the **Role ARN** from its outputs into Ace, and if alerts are on, the **SNS Topic ARN** too.
6. Choose **Connect**.
7. If alerts are on, add Ace to your AMP Alertmanager definition. Ace shows the snippet with your topic ARN and Region:

   ```yaml title="AMP Alertmanager definition"
   alertmanager_config: |
     route:
       receiver: default
       routes:
         - receiver: ace-alerts-sns
           continue: true
     receivers:
       - name: default
       - name: ace-alerts-sns
         sns_configs:
           - topic_arn: "<sns-topic-arn>"
             sigv4:
               region: "<region>"
             subject: "AceAlert"
   ```

   In the Amazon Managed Service for Prometheus console, open **All workspaces** > your workspace > **Alert manager** and choose **Modify**, or **Add definition** if the workspace has none. Merge the `ace-alerts-sns` receiver and route into your existing definition, keeping `continue: true` so alerts still flow to your existing receivers.

**EKS**

1. Choose **In-cluster Prometheus (EKS)**.
2. Under **Capabilities**, **Metrics querying** is always on. Turn on **Alert triggers** if you want Alertmanager alerts to open incidents.
3. Under **Cluster details**, enter the **Cluster region**, **EKS Cluster Name** and **Prometheus Namespace** (default `monitoring`).
4. Choose **Launch stack in AWS**, then review the stack in the AWS console and create it in the account that owns the cluster.
5. When the stack completes, copy the **Role ARN** from its outputs into Ace.
6. Under **Prometheus service**, enter the **Service Name** (for example `prometheus-operated`) and **Port** (for example `9090`).
7. If alerts are on, choose **Generate secret** under **Webhook Secret** and store it, because you can't see it again after you connect.
8. Choose **Connect**.
9. If alerts are on, add Ace as a receiver in your `alertmanager.yaml` or Alertmanager custom resource. Ace shows the snippet with your webhook URL:

   ```yaml title="alertmanager.yaml"
   receivers:
     - name: ace-alerts
       webhook_configs:
         - url: "<webhook-url-from-ace>"
           send_resolved: true
           http_config:
             authorization:
               type: Bearer
               credentials: "<your-webhook-secret>"

   route:
     routes:
       - receiver: ace-alerts
         continue: true
   ```

   Copy the webhook URL Ace shows and use the secret from step 7. Keep `continue: true` so alerts still flow to your existing receivers.

## Confirm it's working

- The panel shows **Validating integration…** while Ace checks access. On AMP, it checks role access and runs a test query, and on EKS, it checks role access, the cluster, and Prometheus build info, targets and rules. With alerts on, it also checks Alertmanager.
- When validation passes, the panel shows **Connected** with the Region.
- With alerts on, the panel shows **Alert triggers — no events received yet** until the first alert arrives, then **Alert triggers active**.

## Disconnect

In Ace, open the connection in **Settings** > **Data sources** > **Prometheus** and choose **Disconnect**. Then:

1. Remove the `ace-alerts-sns` or `ace-alerts` receiver and route from your Alertmanager configuration.
2. Delete the Ace Prometheus stack in CloudFormation, which removes the IAM role along with the SNS topic and subscription or the EKS access entry.

## Next steps

- [Kubernetes](https://tryace.dev/docs/integrations/kubernetes): read pods, events and logs from the same cluster.
- [AWS](https://tryace.dev/docs/integrations/aws): add CloudWatch, CloudTrail and infrastructure metadata.
- [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): how alerts become incidents.