Datadog

Ace reads Datadog metrics, logs, traces and monitors during investigations, and monitor notifications open incidents. If you allow it, Ace also puts the paging monitor into a downtime while an approved fix rolls out.

What Ace readsCopy Anchor Link

During an investigation, Ace reads:

• Metric time series for the incident window, and monitor status.
• Recent error logs.
• APM spans and aggregates (counts, failures, p95 and p99 latency), full traces, the service map, and deployment versions from the version tag.
• Error Tracking issues.

When you connect, Ace discovers your active metrics and monitors, plus the services and environments in monitor tags. With the optional usage_read scope, it also scans for custom metrics nobody queries.

What Ace can changeCopy Anchor Link

If the application key has create_webhooks and monitors_write, Ace creates its own webhook and mentions it in the message of every monitor that can open incidents, as listed below. Ace keeps those mentions current, so new monitors get one and monitors that can't open incidents lose it.

If an admin allows it, Ace puts the paging monitor into a downtime when an approved fix starts, so on-call isn't paged during the rollout. The downtime covers only the alerting group and ends on its own. Downtimes run without approval unless an admin requires one in an action policy, and you can undo each one from the incident. See Actions and approvals.

What opens an incidentCopy Anchor Link

A monitor notification opens an incident, or joins an open one, when its transition is Triggered, Re-Triggered, Warn, Re-Warn, No Data, Re-No Data or Renotify, and Recovered resolves it. Ace groups notifications by monitor and scope.

These monitors never open incidents:

• CI pipeline and CI test monitors.
• Cloud Cost and Audit Trail monitors.
• Synthetic tests tagged ci, ci:true, ci_only, ci-only, env:ci or source:ci.
• Any monitor tagged ace:ignore.

A notification opens an incident only for a workload that routes Datadog. To route it, open the workload's settings, turn on Integration alerts under Investigation triggers and select Datadog.

Ace doesn't poll Datadog for alerts, so until the webhook delivers, Ace uses Datadog only as evidence.

Before you startCopy Anchor Link

• The admin role in Ace.
• A Datadog API key and application key on the US1, US3, US5, EU, AP1, AP2 or US1-FED site. Ace calls Datadog's public API for that site, with no custom endpoint or proxy option.

Credentials and permissionsCopy Anchor Link

Application key scopes:

Select only these scopes, because an unscoped key carries all of its owner's permissions. You can also create the key on a service account with the Datadog Read Only Role, though alert wiring, downtimes and cost scans still need their extra scopes.

Set upCopy Anchor Link

1. In Ace, go to Settings > Data sources > Datadog.
2. Under Credentials, choose your Datadog site and enter the API key and Application key. Create key opens the application keys page for your site.
3. Choose Connect Datadog.
4. Under Alerts, if the key has create_webhooks and monitors_write, Ace sets up the webhook and shows Ace set up a webhook mentioned in every monitor.
5. Otherwise, set it up by hand:

   1. Copy the webhook URL Ace shows.

   2. Choose Generate token and copy the token. Ace shows it once.

   3. In Datadog, open Integrations > Webhooks and add a webhook with that URL. Set Custom Headers to:

      Copy Text

      {"Authorization": "Bearer <token>"}

   4. Mention @webhook-<name> in the message of each monitor that should open incidents.

6. Optional: turn on Allow Ace to update this tool during incidents to let Ace create monitor downtimes during approved fixes.
7. In each workload's settings, select Datadog under Integration alerts.

You can't edit Datadog credentials in place, so to change keys, disconnect and connect again.

Confirm it's workingCopy Anchor Link

• Validation shows API authentication passing, and APM spans, Error Tracking and Usage metering either pass or are skipped with the scope to add.
• The alerts status moves from Waiting for first event to Verified on the first signed delivery. To check right away, send a test notification from a monitor.
• If it shows Last delivery failed because the token didn't match, copy the current token into the webhook's Authorization header.
• With downtimes allowed, the action list shows Datadog monitor downtime as Available. If it shows that it needs monitors_downtime, add that scope.

DisconnectCopy Anchor Link

In Ace, choose Disconnect in Settings > Data sources > Datadog. Ace removes its webhook mention from every monitor, deletes the webhook and then deletes the connection, and anything it can't remove appears as a warning for you to clean up in Datadog.

In Datadog, revoke the application key and API key you created for Ace in Organization Settings. If you wired alerts by hand, delete the webhook and remove the mention from your monitors.

Next stepsCopy Anchor Link

• Actions and approvals: how observability updates and undo work.
• Memory: how Ace learns from your Datadog alerts.
• Integrations: connect more data sources.