Inngest

Failed Inngest runs open incidents in Ace, one incident per function. Once the fix is live, Ace can rerun the failed runs in stages.

What Ace readsCopy Anchor Link

Ace reads Inngest while it investigates a workload that lists it under Investigation sources or routes its alerts.

Ace reads only the environment named in the Environment field and sends it with every request, so a key that covers several environments still reads only this one. An Apps allowlist limits reads to those apps. During an investigation Ace reads:

• Failed runs in the incident window, grouped by function and error, with triggering events, sample run IDs, error messages and in-app stack frames.
• One run's status, timing, trigger, error and step trace. Ace doesn't get inputs or outputs.
• Apps with their latest sync, and each function's triggers, paused state, retries and flow control.
• The backlog of queued, running and paused runs since the incident began, by function.
• Failed step attempts by function and step, from Inngest Insights, whose data lags a few minutes.
• Failed runs of the incident's function after a fix, plus at least one completed run, to confirm the function works again.

Ace also adds your apps and functions to the System Map. Ace blocks any Inngest request beyond these reads and the actions below before it leaves Ace.

What Ace can changeCopy Anchor Link

Ace proposes Inngest actions only after an admin turns on Allow Ace to run approved actions for the connection. Each one needs approval by default, and earned autonomy can let some of them run under an Automatic policy for a workload and environment you confirm. Others always need a person. See Actions and approvals.

A rerun replays every step from the original event, so non-idempotent side effects happen again. Ace rechecks the incident between stages.

These actions need an API key with Write permission. Inngest doesn't let Ace check this in advance, so Settings shows that the permission is checked when the action runs.

What opens an incidentCopy Anchor Link

Failures open incidents only on workloads that route this connection (Integration alerts on, Inngest selected). Ace polls only routed connections.

• Ace polls every few minutes for runs with status FAILED. Each failure opens or joins an incident for its function, at high severity.
• For alerts within seconds, forward Inngest's inngest/function.failed event to Ace (step 4 under Set up). Ace treats a forwarded failure the same as a polled failure of that run.
• Failures open incidents only when the connected environment is production. In any other environment they add context to an open incident.

App syncs join the change ledger as deploys, and function pauses and resumes join it as configuration changes, dated when Ace first sees them. Neither opens an incident.

Before you startCopy Anchor Link

• The admin role in Ace.
• An Inngest organization admin, to create API keys.
• An API key (sk-inn-api-…). Ace refuses signing keys, because a signing key is your app's own secret and can do everything in the environment.

Credentials and permissionsCopy Anchor Link

Set upCopy Anchor Link

1. Create an API key in InngestCopy Anchor Link

1. Open the profile menu > API Keys.
2. Create a key named Ace with Read permission, or Write if Ace should rerun, cancel and pause after approval. Limit the key to the environment Ace reads, which is production unless you name another.
3. Copy the key (sk-inn-api-…).

2. Connect Inngest in AceCopy Anchor Link

1. Open Settings > Data sources and choose Inngest.

2. Fill in the form:

   Field	What to enter
   API key	The key you created.
   Environment	Optional. The environment slug Ace reads. Defaults to production.
   Apps	Optional. A comma-separated allowlist of app IDs (the id you pass to new Inngest({ id })).

3. Select Connect Inngest. Before it saves, Ace checks that the key can read runs and apps in that environment.

3. Route failures to a workloadCopy Anchor Link

Open the workload in Ace, turn on Integration alerts, and select Inngest as a source. Polling starts within a few minutes.

4. Optional: forward failures for faster alertsCopy Anchor Link

1. In the Alerts section, copy the webhook URL.
2. Select Generate token and copy the token, which Ace shows only once. Store the URL and token in your app's environment, for example as ACE_INNGEST_URL and ACE_INNGEST_TOKEN.
3. Add a function triggered by inngest/function.failed to an app in the same environment. It sends the event unchanged as JSON, inside step.run so Inngest retries it.

This example uses the Inngest TypeScript SDK v4. On SDK v3, pass { event: "inngest/function.failed" } as a separate second argument instead of triggers.

import { inngest } from "./client";

export const forwardFailuresToAce = inngest.createFunction(
  { id: "forward-failures-to-ace", triggers: [{ event: "inngest/function.failed" }] },
  async ({ event, step }) => {
    await step.run("send-to-ace", async () => {
      const response = await fetch(process.env.ACE_INNGEST_URL!, {
        method: "POST",
        headers: {
          "content-type": "application/json",
          authorization: `Bearer ${process.env.ACE_INNGEST_TOKEN}`,
        },
        body: JSON.stringify(event),
      });
      if (!response.ok) throw new Error(`Ace returned ${response.status}`);
    });
  },
);

Ace accepts a forwarded failure only if its function and run IDs have Inngest's format and the function is in your Apps allowlist.

5. Optional: allow actionsCopy Anchor Link

Turn on Allow Ace to run approved actions. To let reruns earn Automatic, confirm their workload and environment on the same panel.

Confirm it's workingCopy Anchor Link

• The connection's checks show which environment the key reads and how many apps it sees, and flag an environment that doesn't exist for the key, or an allowlisted app it can't see.
• Once forwarding is set up, the Alerts badge shows Not set up, Waiting for first event, Verified or Last delivery failed. It turns Verified when the first delivery with the right token arrives.
• To rotate the token, select Generate a new token and update your app, since the old token stops working.

DisconnectCopy Anchor Link

1. In Ace, open the Inngest connection under Settings > Data sources and select Disconnect. To stop actions and keep the connection, turn off Allow Ace to run approved actions instead.
2. In Inngest, delete the Ace API key, and remove the forwarding function from your app if you added it. Ace creates nothing in Inngest.

Next stepsCopy Anchor Link

• Actions and approvals
• Trigger.dev
• All integrations