# Honeycomb

> Connect Honeycomb so triggers and SLO burn alerts open incidents in Ace, and Ace investigates them by querying your traces and events.

Source: https://tryace.dev/docs/integrations/honeycomb

Ace queries each service's traces and events in Honeycomb against a baseline and reads SLOs, triggers and markers. Triggers and SLO burn alerts open incidents, and deploy markers join the workload's change timeline.

| | |
| --- | --- |
| Category | APM |
| Ace reads | Columns, query results, traces, SLOs, triggers, markers |
| Ace can change | Markers on the environment when an incident opens and resolves |
| Opens incidents | Yes, from triggers and SLO burn alerts |
| Setup | Settings form |

## What Ace reads

Ace reads only the datasets you list, plus triggers, SLOs and markers that apply to the whole environment:

- A dataset's columns, before it queries the dataset.
- Queries over the incident window: counts, averages, percentiles, heatmaps, distinct counts and rates.
- Comparisons of failing events against the rest, to find the route, version, host or customer they share.
- Traces, one at a time.
- SLOs and their burn, triggers and whether they fired, and markers around the incident.

Queries, comparisons and traces need **Run Queries**, which Honeycomb offers on Enterprise. Without it, Ace still reads triggers, SLOs and markers.

After a fix, Ace queries the incident's dataset to confirm the errors stopped. For the System Map, it discovers your datasets and, on Enterprise, the service map's dependencies and the routes each service serves.

## What Ace can change

| Action | What it does | Approval | Undo |
| --- | --- | --- | --- |
| Marker | When an incident opens or resolves, Ace adds one environment-wide marker so your charts show it, for example "Incident opened: `<headline>` (workload `<name>`)" | Runs on its own | Yes. Undo from the incident deletes the marker |

Ace adds markers only after an admin turns on **Allow Ace to update this tool during incidents** for the connection, and only when the key has **Manage Markers**. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals).

## What opens an incident

Triggers and burn alerts reach Ace through a webhook recipient as they fire. While Honeycomb is routed to a workload, Ace also polls every 2 minutes for fired triggers and burn alerts in the environment and your datasets, so a missed delivery still opens an incident.

- A **TRIGGERED** notification opens an incident or joins the one already open on the workload, and **OK** resolves it.
- An SLO resolves once none of its burn alerts is still triggered.
- Ace ignores test notifications.
- Ace ignores alerts that name only datasets you didn't list, but accepts alerts for the whole environment.
- To keep a trigger, SLO or burn alert out of Ace, such as one for an audit or business query, put `ace:ignore` in its name or description.

Severity comes from a `severity`, `priority` or `sev` tag, and without one, Ace sets it from the alert itself. Deploy markers on the environment and your datasets go to the workload's change timeline and never open an incident.

An alert opens an incident only when Honeycomb is routed to a workload. To route it, open the workload's settings, turn on **Integration alerts** under **Investigation triggers** and select Honeycomb under **Alert sources**.

## Before you start

- The **admin** role in Ace. See [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions).
- The **Team Owner** role in Honeycomb, to create a configuration key.
- Honeycomb Enterprise for queries, comparisons, traces and service map discovery.

## Credentials and permissions

Ace uses one configuration key for one environment. Ingest keys don't work.

| Credential | Where to create it | Permissions | Why Ace needs it |
| --- | --- | --- | --- |
| Configuration key (required) | **Manage Environments** > the environment > **API Keys** > **Configuration** | **Manage Queries and Columns**, **Run Queries** (Enterprise), **Manage SLOs**, **Manage Triggers**, **Manage Markers** | Reads columns, runs queries, and reads SLOs, triggers and markers |
| Same key (only for alerts) | | **Manage Recipients** | Adds Ace's webhook recipient to triggers and burn alerts. Recipients are team-wide |
| Same key (only for actions) | | **Manage Markers** | Adds and deletes incident markers |

## Set up

### Create the configuration key

1. As a Team Owner, open the Environments selector > **Manage Environments** > the environment > **API Keys** > **Configuration** > **Create Configuration API Key**.
2. Name it Ace and select **Manage Queries and Columns**, **Run Queries**, **Manage SLOs**, **Manage Triggers** and **Manage Markers**. Add **Manage Recipients** if Ace should wire alerts, knowing that this permission changes recipients for the whole team.
3. Create the key and copy it.

### Connect Honeycomb in Ace

1. In Ace, go to **Settings** > **Data sources** > **Honeycomb**.
2. Fill in the form:
   - **Configuration key**: the key you copied.
   - **Region**: **US (api.honeycomb.io)** or **EU (api.eu1.honeycomb.io)**.
   - **Datasets**: the datasets (services) Ace may query, alert on and mark, comma-separated.
3. Choose **Connect Honeycomb**. If any validation check fails, Ace doesn't save the key.

### Wire alerts

If the key has **Manage Recipients**, Ace adds a webhook recipient to every trigger and burn alert of the environment and your datasets, skipping those marked `ace:ignore`. Ace checks again every hour to pick up new triggers, and the **Alerts** section shows what Ace set up.

To set it up yourself, choose **Show manual setup** in the **Alerts** section:

1. Copy the webhook URL Ace shows.
2. Choose **Generate token** and copy the token. Ace shows it once.
3. In Honeycomb, open **Team Settings** > **Integrations**. Under **Trigger, SLOs, and Anomaly Detection Recipients**, choose **Add Integration**, set **Provider** to Webhook, enter the URL and the token as its **Shared Secret**, and choose **Add**.
4. Add the recipient to the triggers and burn alerts that should open incidents.

### Allow markers (optional)

On the connected Honeycomb page, turn on **Allow Ace to update this tool during incidents**.

## Confirm it's working

- The connection shows **Connected** after Ace checks that the key is a configuration key with each permission it needs and that it can find each dataset you listed.
- The **Alerts** section shows **Waiting for first event** until the first delivery Ace accepts, then **Verified**.

## Disconnect

1. In Ace, go to **Settings** > **Data sources** > **Honeycomb** and choose **Disconnect**. Ace removes its recipient from your triggers and burn alerts and deletes it, except where a burn alert notifies only Ace, in which case Ace leaves the recipient in place. When Ace can't remove something, it tells you so you can remove `ace-<id>` in Honeycomb yourself.
2. In Honeycomb, delete the configuration key.

## Next steps

- [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): how observability updates and undo work.
- [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): how alerts become incidents.
- [Integrations](https://tryace.dev/docs/integrations): connect more data sources.