New Relic
Connect New Relic so alert issues open incidents in Ace, and Ace investigates them with your APM, OpenTelemetry, trace, error and NRQL data.
Ace reads golden metrics, traces, dependencies, Errors Inbox and NRQL from New Relic during investigations. Alert issues open incidents, and change-tracking deployments join the workload's change timeline.
| Category | APM |
| Ace reads | Alert issues, golden metrics, distributed traces, related entities, Errors Inbox, change-tracking deployments, NRQL results |
| Ace can change | Muting rules for the condition that paged, while an approved fix rolls out |
| Opens incidents | Yes, from alert issues |
| Setup | Settings form |
What Ace reads
Ace reads through the NerdGraph API, and only in the accounts you list:
- Alert issues around the incident, with title, priority, state, condition, policy, entities and the incidents inside each issue.
- An entity's golden metrics, compared with the window before the incident.
- Related entities: callers, downstream services, databases and hosts.
- Distributed traces, one at a time.
- Errors Inbox error groups, when your key can read them. New Relic's API doesn't return APM stack traces.
- Change-tracking deployments before and during the incident.
- NRQL, one query at a time over the incident window, including comparisons with an earlier period.
After a fix, Ace checks the entity's error rate and latency. For the System Map, it discovers the APM applications and OpenTelemetry services in your accounts and maps the calls between them, along with the databases, external hosts and hosts they connect to.
What Ace can change
| Action | What it does | Approval | Undo |
|---|---|---|---|
| Muting rule | When an approved fix starts rolling out, Ace adds a muting rule named "Ace: fix rolling out" for each condition that paged, for a bounded window. The rule covers only that condition, never the whole entity, and ends on its own | Runs on its own | Yes. Undo from the incident removes the rule |
This is the only change Ace makes in New Relic, and it makes it only after an admin turns on Allow Ace to update this tool during incidents for the connection. See Actions and approvals.
To mute a condition, Ace needs its conditionFamilyId and the entity GUIDs in the alert payload. The workflow Ace creates sends them, but Ace can't mute alerts from a manual workflow that leaves them out.
What opens an incident
Alert issues reach Ace through a workflow webhook as they fire. Ace also polls New Relic every 2 minutes, so a missed delivery still opens an incident.
- An activated or created issue opens an incident or joins the one already open on the workload, and a closed issue resolves it.
- Ace ignores acknowledgements and priority changes.
- A fully muted issue never opens an incident, though a partially muted one does. A muted close doesn't count as recovery.
- Ace ignores issues from accounts you didn't list.
- To keep an issue out of Ace, put
ace:ignorein its title, condition name or policy name.
Severity comes from the issue's priority of Critical, High, Medium or Low. Change-tracking deployments go to the workload's change timeline and never open an incident.
An issue opens an incident only when New Relic is routed to a workload. To route it, open the workload's settings, turn on Integration alerts under Investigation triggers and select New Relic under Alert sources.
Before you start
- The admin role in Ace. See Roles and permissions.
- Permission in New Relic to create a user and grant it roles on each account.
- For alert wiring and muting, a custom role for Ace's user, which needs New Relic Pro or Enterprise.
Credentials and permissions
Ace uses one user key. A user key has no scopes and acts as its user, so create a dedicated user for Ace with only the roles below.
| Credential | Where to create it | Permissions | Why Ace needs it |
|---|---|---|---|
| User key (required) | API keys in New Relic, signed in as Ace's user | Read access to each account, and Applied intelligence > Issues: view | Reads telemetry and alert issues |
| Same key (only for alerts) | Applied intelligence > Destinations: modify, Channels: modify and delete, Workflows: modify and delete | Adds and maintains Ace's webhook destination, channel and workflow | |
| Same key (only for actions) | Alerts > Lifecycle overrides (muting rules): modify and delete | Creates and removes muting rules |
Set up
Create the user key
- In New Relic, create a user for Ace and grant it the standard Read only role on each account Ace should read, in Administration > Access management > Access grants.
- For alert wiring and muting, add a custom role in Roles > Add a role with Applied intelligence > Destinations, Channels and Workflows, and Alerts > Lifecycle overrides, then grant it to Ace's user.
- Signed in as Ace's user, open API keys (
https://one.newrelic.com/api-keys, orhttps://one.eu.newrelic.com/api-keysin the EU) and choose Create a key. Set Key type to User, name it Ace and save it. - Copy the key from the row's menu.
Connect New Relic in Ace
- In Ace, go to Settings > Data sources > New Relic.
- Fill in the form:
- User key: the key you copied. It starts with
NRAK-. - Data center: US (api.newrelic.com) or EU (api.eu.newrelic.com).
- Account IDs: the accounts Ace may read, alert on and write to, comma-separated. Each ID is in the account picker at the top of New Relic, or under Administration > Access management > Accounts.
- User key: the key you copied. It starts with
- Choose Connect New Relic. If any validation check fails, Ace doesn't save the key.
Wire alerts
If the key can manage destinations and workflows, Ace sets up a webhook destination, channel and workflow named ace-<id> in each account, for up to 20 accounts. The workflow covers every issue.
The Alerts section shows what Ace set up, and Ace checks it again every hour and removes it when you disconnect.
To set it up yourself, choose Show manual setup in the Alerts section:
- Copy the webhook URL Ace shows.
- Choose Generate token and copy the token. Ace shows it once.
- In New Relic, add a Webhook destination with the URL and Bearer token authorization.
- Add a workflow that notifies the destination when issues are activated and closed.
- Give the workflow a JSON payload with these fields:
issueId,title,state,priority,issuePageUrl,triggerEventandmutingState.accountId, set to thenrAccountIdvariable.conditionName,conditionFamilyIdandpolicyNamefrom accumulations.entityNames,entityGuidsandentityTypesfromentitiesData.
Allow muting (optional)
On the connected New Relic page, turn on Allow Ace to update this tool during incidents.
Confirm it's working
- The connection shows Connected after Ace checks that the key is a user key and sees each account in the data center you chose, runs a test NRQL query and reads issues.
- The Alerts section shows Waiting for first event until the first delivery Ace accepts, then Verified.
Disconnect
- In Ace, go to Settings > Data sources > New Relic and choose Disconnect. Ace removes what it set up in each account, and if it can't, it tells you so you can remove
ace-<id>in New Relic yourself. - In New Relic, delete Ace's user key, or the Ace user.
Next steps
- Actions and approvals: how observability updates and undo work.
- How Ace works: how alerts become incidents.
- Integrations: connect more data sources.