# New Relic

> Connect New Relic so alert issues open incidents in Ace, and Ace investigates them with your APM, OpenTelemetry, trace, error and NRQL data.

Source: https://tryace.dev/docs/integrations/new-relic

Ace reads golden metrics, traces, dependencies, Errors Inbox and NRQL from New Relic during investigations. Alert issues open incidents, and change-tracking deployments join the workload's change timeline.

| | |
| --- | --- |
| Category | APM |
| Ace reads | Alert issues, golden metrics, distributed traces, related entities, Errors Inbox, change-tracking deployments, NRQL results |
| Ace can change | Muting rules for the condition that paged, while an approved fix rolls out |
| Opens incidents | Yes, from alert issues |
| Setup | Settings form |

## What Ace reads

Ace reads through the NerdGraph API, and only in the accounts you list:

- Alert issues around the incident, with title, priority, state, condition, policy, entities and the incidents inside each issue.
- An entity's golden metrics, compared with the window before the incident.
- Related entities: callers, downstream services, databases and hosts.
- Distributed traces, one at a time.
- Errors Inbox error groups, when your key can read them. New Relic's API doesn't return APM stack traces.
- Change-tracking deployments before and during the incident.
- NRQL, one query at a time over the incident window, including comparisons with an earlier period.

After a fix, Ace checks the entity's error rate and latency. For the System Map, it discovers the APM applications and OpenTelemetry services in your accounts and maps the calls between them, along with the databases, external hosts and hosts they connect to.

## What Ace can change

| Action | What it does | Approval | Undo |
| --- | --- | --- | --- |
| Muting rule | When an approved fix starts rolling out, Ace adds a muting rule named "Ace: fix rolling out" for each condition that paged, for a bounded window. The rule covers only that condition, never the whole entity, and ends on its own | Runs on its own | Yes. Undo from the incident removes the rule |

This is the only change Ace makes in New Relic, and it makes it only after an admin turns on **Allow Ace to update this tool during incidents** for the connection. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals).

To mute a condition, Ace needs its `conditionFamilyId` and the entity GUIDs in the alert payload. The workflow Ace creates sends them, but Ace can't mute alerts from a manual workflow that leaves them out.

## What opens an incident

Alert issues reach Ace through a workflow webhook as they fire. Ace also polls New Relic every 2 minutes, so a missed delivery still opens an incident.

- An **activated** or **created** issue opens an incident or joins the one already open on the workload, and a **closed** issue resolves it.
- Ace ignores acknowledgements and priority changes.
- A **fully muted** issue never opens an incident, though a partially muted one does. A muted close doesn't count as recovery.
- Ace ignores issues from accounts you didn't list.
- To keep an issue out of Ace, put `ace:ignore` in its title, condition name or policy name.

Severity comes from the issue's priority of Critical, High, Medium or Low. Change-tracking deployments go to the workload's change timeline and never open an incident.

An issue opens an incident only when New Relic is routed to a workload. To route it, open the workload's settings, turn on **Integration alerts** under **Investigation triggers** and select New Relic under **Alert sources**.

## Before you start

- The **admin** role in Ace. See [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions).
- Permission in New Relic to create a user and grant it roles on each account.
- For alert wiring and muting, a custom role for Ace's user, which needs New Relic Pro or Enterprise.

## Credentials and permissions

Ace uses one user key. A user key has no scopes and acts as its user, so create a dedicated user for Ace with only the roles below.

| Credential | Where to create it | Permissions | Why Ace needs it |
| --- | --- | --- | --- |
| User key (required) | **API keys** in New Relic, signed in as Ace's user | Read access to each account, and **Applied intelligence** > **Issues**: view | Reads telemetry and alert issues |
| Same key (only for alerts) | | **Applied intelligence** > **Destinations**: modify, **Channels**: modify and delete, **Workflows**: modify and delete | Adds and maintains Ace's webhook destination, channel and workflow |
| Same key (only for actions) | | **Alerts** > **Lifecycle overrides** (muting rules): modify and delete | Creates and removes muting rules |

## Set up

### Create the user key

1. In New Relic, create a user for Ace and grant it the standard **Read only** role on each account Ace should read, in **Administration** > **Access management** > **Access grants**.
2. For alert wiring and muting, add a custom role in **Roles** > **Add a role** with **Applied intelligence** > **Destinations**, **Channels** and **Workflows**, and **Alerts** > **Lifecycle overrides**, then grant it to Ace's user.
3. Signed in as Ace's user, open **API keys** (`https://one.newrelic.com/api-keys`, or `https://one.eu.newrelic.com/api-keys` in the EU) and choose **Create a key**. Set **Key type** to User, name it Ace and save it.
4. Copy the key from the row's menu.

### Connect New Relic in Ace

1. In Ace, go to **Settings** > **Data sources** > **New Relic**.
2. Fill in the form:
   - **User key**: the key you copied. It starts with `NRAK-`.
   - **Data center**: **US (api.newrelic.com)** or **EU (api.eu.newrelic.com)**.
   - **Account IDs**: the accounts Ace may read, alert on and write to, comma-separated. Each ID is in the account picker at the top of New Relic, or under **Administration** > **Access management** > **Accounts**.
3. Choose **Connect New Relic**. If any validation check fails, Ace doesn't save the key.

### Wire alerts

If the key can manage destinations and workflows, Ace sets up a webhook destination, channel and workflow named `ace-<id>` in each account, for up to 20 accounts. The workflow covers every issue.

The **Alerts** section shows what Ace set up, and Ace checks it again every hour and removes it when you disconnect.

To set it up yourself, choose **Show manual setup** in the **Alerts** section:

1. Copy the webhook URL Ace shows.
2. Choose **Generate token** and copy the token. Ace shows it once.
3. In New Relic, add a Webhook destination with the URL and Bearer token authorization.
4. Add a workflow that notifies the destination when issues are activated and closed.
5. Give the workflow a JSON payload with these fields:
   - `issueId`, `title`, `state`, `priority`, `issuePageUrl`, `triggerEvent` and `mutingState`.
   - `accountId`, set to the `nrAccountId` variable.
   - `conditionName`, `conditionFamilyId` and `policyName` from accumulations.
   - `entityNames`, `entityGuids` and `entityTypes` from `entitiesData`.

### Allow muting (optional)

On the connected New Relic page, turn on **Allow Ace to update this tool during incidents**.

## Confirm it's working

- The connection shows **Connected** after Ace checks that the key is a user key and sees each account in the data center you chose, runs a test NRQL query and reads issues.
- The **Alerts** section shows **Waiting for first event** until the first delivery Ace accepts, then **Verified**.

## Disconnect

1. In Ace, go to **Settings** > **Data sources** > **New Relic** and choose **Disconnect**. Ace removes what it set up in each account, and if it can't, it tells you so you can remove `ace-<id>` in New Relic yourself.
2. In New Relic, delete Ace's user key, or the Ace user.

## Next steps

- [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): how observability updates and undo work.
- [How Ace works](https://tryace.dev/docs/concepts/how-ace-works): how alerts become incidents.
- [Integrations](https://tryace.dev/docs/integrations): connect more data sources.