# Inngest

> Connect Inngest so failed production runs open incidents, app syncs join the change ledger, and Ace can rerun, cancel or pause after approval.

Source: https://tryace.dev/docs/integrations/inngest

Failed Inngest runs open incidents in Ace, one incident per function. Once the fix is live, Ace can rerun the failed runs in stages.

| | |
| --- | --- |
| Category | Jobs |
| Ace reads | Failed runs, run details and step traces, apps and functions, backlog, step failures from Insights |
| Ace can change | Rerun failed runs, cancel runs, pause or resume a function, only after an admin allows actions |
| Opens incidents | Yes, for failed production runs of workloads that route Inngest |
| Setup | Settings form, plus an optional forwarding function in your app |

## What Ace reads

Ace reads Inngest while it investigates a workload that lists it under **Investigation sources** or routes its alerts.

Ace reads only the environment named in the **Environment** field and sends it with every request, so a key that covers several environments still reads only this one. An **Apps** allowlist limits reads to those apps. During an investigation Ace reads:

- Failed runs in the incident window, grouped by function and error, with triggering events, sample run IDs, error messages and in-app stack frames.
- One run's status, timing, trigger, error and step trace. Ace doesn't get inputs or outputs.
- Apps with their latest sync, and each function's triggers, paused state, retries and flow control.
- The backlog of queued, running and paused runs since the incident began, by function.
- Failed step attempts by function and step, from Inngest Insights, whose data lags a few minutes.
- Failed runs of the incident's function after a fix, plus at least one completed run, to confirm the function works again.

Ace also adds your apps and functions to the System Map. Ace blocks any Inngest request beyond these reads and the actions below before it leaves Ace.

## What Ace can change

Ace proposes Inngest actions only after an admin turns on **Allow Ace to run approved actions** for the connection. Each one needs approval by default, and earned autonomy can let some of them run under an Automatic policy for a workload and environment you confirm. Others always need a person. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals).

| Action | What it does | Approval | Undo |
| --- | --- | --- | --- |
| Run rerun | Reruns the failed runs of one function from the incident window in stages, for example 10 first and then the rest, up to 500 runs. Each run reruns exactly once. It waits for the incident's fix to deploy before it starts. | Approval, or Automatic under your policy | No |
| Function pause | Pauses one function so no new runs start. Use it as a kill switch. Inngest cancels runs still unfinished after 7 days paused. | Always a person | Yes. Resumes the function. |
| Function resume | Resumes a paused function. | Always a person | Yes |
| Run cancel | Cancels one function's queued, running and paused runs from the incident window in stages, up to 500 runs. Each run is cancelled exactly once. | Always a person | No. A cancelled run can't be resumed. |

A rerun replays every step from the original event, so non-idempotent side effects happen again. Ace rechecks the incident between stages.

These actions need an API key with **Write** permission. Inngest doesn't let Ace check this in advance, so Settings shows that the permission is checked when the action runs.

## What opens an incident

Failures open incidents only on workloads that route this connection (**Integration alerts** on, Inngest selected). Ace polls only routed connections.

- Ace polls every few minutes for runs with status `FAILED`. Each failure opens or joins an incident for its function, at high severity.
- For alerts within seconds, forward Inngest's `inngest/function.failed` event to Ace (step 4 under [Set up](#set-up)). Ace treats a forwarded failure the same as a polled failure of that run.
- Failures open incidents only when the connected environment is production. In any other environment they add context to an open incident.

App syncs join the change ledger as deploys, and function pauses and resumes join it as configuration changes, dated when Ace first sees them. Neither opens an incident.

## Before you start

- The **admin** role in Ace.
- An Inngest **organization admin**, to create API keys.
- An API key (`sk-inn-api-…`). Ace refuses signing keys, because a signing key is your app's own secret and can do everything in the environment.

## Credentials and permissions

| Credential | Where to create it | Permission | Why Ace needs it |
| --- | --- | --- | --- |
| API key (needed) | Profile menu > **API Keys** | Read, limited to the environment Ace reads | Read runs, traces, apps, functions and Insights. |
| Same key (only for actions) | Same | Write | Rerun and cancel runs, and pause or resume functions. |
| Token (only for forwarded failures) | Generated by Ace | None | Proves that a forwarded failure comes from your app. |

## Set up

### 1. Create an API key in Inngest

1. Open the profile menu > **API Keys**.
2. Create a key named Ace with **Read** permission, or **Write** if Ace should rerun, cancel and pause after approval. Limit the key to the environment Ace reads, which is production unless you name another.
3. Copy the key (`sk-inn-api-…`).

### 2. Connect Inngest in Ace

1. Open **Settings** > **Data sources** and choose **Inngest**.
2. Fill in the form:

   | Field | What to enter |
   | --- | --- |
   | **API key** | The key you created. |
   | **Environment** | Optional. The environment slug Ace reads. Defaults to `production`. |
   | **Apps** | Optional. A comma-separated allowlist of app IDs (the `id` you pass to `new Inngest({ id })`). |

3. Select **Connect Inngest**. Before it saves, Ace checks that the key can read runs and apps in that environment.

### 3. Route failures to a workload

Open the workload in Ace, turn on **Integration alerts**, and select Inngest as a source. Polling starts within a few minutes.

### 4. Optional: forward failures for faster alerts

1. In the **Alerts** section, copy the webhook URL.
2. Select **Generate token** and copy the token, which Ace shows only once. Store the URL and token in your app's environment, for example as `ACE_INNGEST_URL` and `ACE_INNGEST_TOKEN`.
3. Add a function triggered by `inngest/function.failed` to an app in the same environment. It sends the event unchanged as JSON, inside `step.run` so Inngest retries it.

This example uses the Inngest TypeScript SDK v4. On SDK v3, pass `{ event: "inngest/function.failed" }` as a separate second argument instead of `triggers`.

```ts title="forward-failures-to-ace.ts"

  { id: "forward-failures-to-ace", triggers: [{ event: "inngest/function.failed" }] },
  async ({ event, step }) => {
    await step.run("send-to-ace", async () => {
      const response = await fetch(process.env.ACE_INNGEST_URL!, {
        method: "POST",
        headers: {
          "content-type": "application/json",
          authorization: `Bearer ${process.env.ACE_INNGEST_TOKEN}`,
        },
        body: JSON.stringify(event),
      });
      if (!response.ok) throw new Error(`Ace returned ${response.status}`);
    });
  },
);
```

Ace accepts a forwarded failure only if its function and run IDs have Inngest's format and the function is in your **Apps** allowlist.

### 5. Optional: allow actions

Turn on **Allow Ace to run approved actions**. To let reruns earn Automatic, confirm their workload and environment on the same panel.

## Confirm it's working

- The connection's checks show which environment the key reads and how many apps it sees, and flag an environment that doesn't exist for the key, or an allowlisted app it can't see.
- Once forwarding is set up, the **Alerts** badge shows **Not set up**, **Waiting for first event**, **Verified** or **Last delivery failed**. It turns **Verified** when the first delivery with the right token arrives.
- To rotate the token, select **Generate a new token** and update your app, since the old token stops working.

## Disconnect

1. In Ace, open the Inngest connection under **Settings** > **Data sources** and select **Disconnect**. To stop actions and keep the connection, turn off **Allow Ace to run approved actions** instead.
2. In Inngest, delete the Ace API key, and remove the forwarding function from your app if you added it. Ace creates nothing in Inngest.

## Next steps

- [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals)
- [Trigger.dev](https://tryace.dev/docs/integrations/trigger-dev)
- [All integrations](https://tryace.dev/docs/integrations)