Prometheus

Connect Amazon Managed Service for Prometheus or an in-cluster Prometheus on Amazon EKS so Ace can query metrics during investigations and open incidents from Alertmanager alerts.

Ace queries Prometheus for the affected workload's latency, error rate, throttling, memory and restarts, and with Alertmanager connected, your alerts open incidents. Ace supports two deployment types:

Amazon Managed Prometheus (AMP). Ace queries the AMP API, and alerts arrive through an Amazon SNS topic.
In-cluster Prometheus (EKS). Prometheus Operator or kube-prometheus-stack in an Amazon EKS cluster, which Ace queries through the cluster's Kubernetes API server. Alerts arrive through an Alertmanager webhook.

Other self-hosted Prometheus servers aren't supported directly, but you can remote-write their metrics to an AMP workspace and connect that.


Category	Kubernetes metrics and alerts
Ace reads	PromQL queries, scrape targets, rules and metric metadata. With alerts on, Alertmanager status
Ace can change	Nothing (read-only)
Opens incidents	Yes, from Alertmanager alerts, if you turn on alert triggers
Setup	Guided setup in Ace that creates a role in your AWS account

What Ace reads

During an investigation, Ace runs PromQL range queries over the incident window for p95 latency, request rate, error rate, CPU throttling, memory pressure and usage, restart rate and OOM kills. It also reads scrape targets, rules and metric metadata.

On AMP, Ace queries your workspace through the AMP API. On EKS, it queries the Prometheus service through the Kubernetes API server's service proxy and can read resources only in the Prometheus namespace you choose.

What Ace can change

Ace changes nothing in Prometheus. Its role can't change rules, scrape configuration or your Alertmanager definition, and on EKS it also can't deploy workloads, edit manifests or change RBAC.

What opens an incident

Only Alertmanager alerts open incidents, and only if you turn on Alert triggers during setup.

A firing alert opens an incident or joins an open one.
A resolved alert resolves the matching incident.
Alerts named Watchdog or InfoInhibitor, and alerts with severity info or none, never open incidents.

An alert opens an incident only for a workload that routes it. To route it, open the workload's settings, turn on Integration alerts under Investigation triggers and select Alertmanager via AMP or Alertmanager via EKS Prometheus Operator.

Ace doesn't poll Prometheus for alerts, so without Alertmanager alerts, Ace uses Prometheus only as evidence.

Before you start

The admin role in Ace.
Permission in AWS to create CloudFormation stacks and IAM roles.
AMP: an AMP workspace and, for alerts, an AMP Alertmanager definition you can edit. The alerts stack creates an SNS topic, billed at standard Amazon SNS pricing.
EKS: an EKS cluster whose authentication mode allows access entries, with Prometheus running in one namespace, and, for alerts, access to your Alertmanager configuration.

Settings shows one Prometheus connection, either AMP or EKS, so to change its settings, disconnect and connect again.

Private networks

AMP: Ace queries the AMP API with the role the setup creates and alerts arrive through Amazon SNS, so nothing in your VPC needs to be reachable. For a private EKS cluster, remote-write from your in-cluster Prometheus to AMP and connect the workspace.
EKS: Ace connects from its cloud to your cluster's Kubernetes API endpoint, so the cluster needs public endpoint access, and a cluster with only private endpoint access must use AMP. For alerts, Alertmanager needs outbound HTTPS to Ace.

The Kubernetes connector runs inside the cluster and connects out to Ace, so it works with a private API endpoint.

Credentials and permissions

Ace never stores AWS access keys. The guided setup uses CloudFormation to create an IAM role, which Ace assumes with an external ID generated for your workspace, and you review the stack in the AWS console before you create it.

Deployment type	What the setup creates	Why Ace needs it
AMP	A role with read-only query access to your workspace only: metrics, series, labels, metric metadata and rules	Metrics queries and rules
AMP, alerts on	Read access to Alertmanager status, and an SNS topic that only AMP can publish to, from your workspace. An HTTPS subscription delivers to Ace	Alertmanager status and alert delivery
EKS	A role that can describe your cluster only, and an EKS access entry with the AWS-managed `AmazonEKSAdminViewPolicy`, scoped to the Prometheus namespace	Read-only access in that namespace, including the service proxy used to query Prometheus and Alertmanager
EKS, alerts on	A webhook secret generated by Ace. Alertmanager sends it as a bearer token	Authenticates alert deliveries

Set up

In Ace, go to Settings > Data sources > Prometheus and choose your deployment type.

Choose Amazon Managed Prometheus (AMP).
Under Capabilities, Metrics querying is always on. Turn on Alert triggers if you want Alertmanager alerts to open incidents.
Under Configuration, choose the Region and enter the AMP Workspace ID (ws-...).
Choose Launch stack in AWS, then review the stack in the AWS console and create it.
When the stack completes, copy the Role ARN from its outputs into Ace, and if alerts are on, the SNS Topic ARN too.
Choose Connect.
If alerts are on, add Ace to your AMP Alertmanager definition. Ace shows the snippet with your topic ARN and Region:
AMP Alertmanager definition
```
alertmanager_config: |
  route:
    receiver: default
    routes:
      - receiver: ace-alerts-sns
        continue: true
  receivers:
    - name: default
    - name: ace-alerts-sns
      sns_configs:
        - topic_arn: "<sns-topic-arn>"
          sigv4:
            region: "<region>"
          subject: "AceAlert"
```
In the Amazon Managed Service for Prometheus console, open All workspaces > your workspace > Alert manager and choose Modify, or Add definition if the workspace has none. Merge the ace-alerts-sns receiver and route into your existing definition, keeping continue: true so alerts still flow to your existing receivers.

Confirm it's working

The panel shows Validating integration… while Ace checks access. On AMP, it checks role access and runs a test query, and on EKS, it checks role access, the cluster, and Prometheus build info, targets and rules. With alerts on, it also checks Alertmanager.
When validation passes, the panel shows Connected with the Region.
With alerts on, the panel shows Alert triggers — no events received yet until the first alert arrives, then Alert triggers active.

Disconnect

In Ace, open the connection in Settings > Data sources > Prometheus and choose Disconnect. Then:

Remove the ace-alerts-sns or ace-alerts receiver and route from your Alertmanager configuration.
Delete the Ace Prometheus stack in CloudFormation, which removes the IAM role along with the SNS topic and subscription or the EKS access entry.

Next steps

Kubernetes: read pods, events and logs from the same cluster.
AWS: add CloudWatch, CloudTrail and infrastructure metadata.
How Ace works: how alerts become incidents.

What Ace readsCopy Anchor Link

What Ace can changeCopy Anchor Link

What opens an incidentCopy Anchor Link

Before you startCopy Anchor Link

Private networksCopy Anchor Link

Credentials and permissionsCopy Anchor Link

Set upCopy Anchor Link

Confirm it's workingCopy Anchor Link

DisconnectCopy Anchor Link

Next stepsCopy Anchor Link

On this page