# Kubernetes

> Connect a Kubernetes cluster so Ace can read resources, events and logs in the namespaces you choose, open incidents from failing pods and nodes and, if you allow it, make approved changes.

Source: https://tryace.dev/docs/integrations/kubernetes

Ace reads pods, rollouts, events and logs from your cluster during an investigation. The connector runs as Deployments in your cluster, with a read-only reader and an optional, separate writer for approved changes.

Both connect out to Ace over HTTPS, so you don't open an inbound port, expose the Kubernetes API or upload a kubeconfig, and Kubernetes credentials stay in the cluster.

| | |
| --- | --- |
| Category | Cluster resources, events, logs and governed remediation |
| Ace reads | Resources, events and container logs in the namespaces you choose. Optionally nodes and other cluster-wide resources |
| Ace can change | Nothing with the read connector. With a separate write connector, rollouts, scaling, resources and more, after approval |
| Opens incidents | Yes, from failing pods, rollouts and nodes |
| Setup | Connector Deployments in your cluster, installed with commands Ace generates |

## What Ace reads

In each namespace you choose, the reader can `get` and `list` these resources and read current and previous container logs:

- Pods, Services, Endpoints, EndpointSlices, ConfigMaps, Events, PersistentVolumeClaims, ResourceQuotas, LimitRanges, ServiceAccounts
- Deployments, ReplicaSets, StatefulSets, DaemonSets, ControllerRevisions, Jobs, CronJobs
- Ingresses, NetworkPolicies, HorizontalPodAutoscalers, PodDisruptionBudgets, Roles, RoleBindings
- Argo CD Applications, Argo Rollouts, Flux Kustomizations and HelmReleases, when your cluster serves them

The reader never reads or writes Secrets.

**Read cluster-wide resources such as nodes and admission webhooks** adds Nodes, Namespaces, PersistentVolumes, StorageClasses, ClusterRoles, ClusterRoleBindings, and mutating and validating webhook configurations. Without it, the only thing the reader reads outside your chosen namespaces is the `kube-system` namespace, which it uses to identify the cluster. An investigation reads only the namespaces mapped to the incident's workload.

## What Ace can change

The reader can't change anything. For changes, install a separate write connector and confirm which namespaces it can act on for each workload and environment, and Ace can then propose:

- Rollouts: restart, undo, pause and resume, and set a container image.
- Capacity: scale, update an autoscaler, set container resources, update a disruption budget.
- Jobs: suspend a CronJob. Resume a CronJob or re-run a Job (approval only).
- Configuration: set a non-secret environment value (approval only).
- Pods and nodes: delete a pod, and cordon, uncordon, taint, untaint or drain a node. These are all approval only, and node operations need cluster administration.
- GitOps controllers: Argo Rollouts abort, undo, promote and retry, Argo CD rollback, sync and auto-sync on or off, and Flux reconcile, suspend and resume.
- An exact patch, create or delete on a supported resource.

A person approves each exact operation unless an admin grants automatic repairs, and approval-only operations never run automatically.

Most operations have an **Undo**, including scale, rollout undo, and image and resource changes. A rollout restart, pod delete, Job re-run and node drain can't be undone, and undoing a drain only uncordons the node.

Automatic repairs are how Earned autonomy works in Kubernetes, and they need their own grant. An admin names the exact resource, field and allowed value, sets an expiry of at most seven days, a daily limit and a recovery window, and then confirms **I authorize automatic restoration of these fields and values for this workload and environment.**

Ace repairs automatically only within that grant and only after it delivers a countdown to Slack, so your team sees the repair coming. If a repair makes things worse, automatic repairs stop. See [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals).

## What opens an incident

The reader watches pod, rollout and node state in the mapped namespaces:

| Condition | Severity |
| --- | --- |
| `CrashLoopBackOff` | Critical |
| `ImagePullBackOff` or `ErrImagePull` | Critical |
| Deployment `ProgressDeadlineExceeded` | Critical |
| Node `NotReady` | Critical |
| Repeated `OOMKilled` restarts | High |
| Recent pod eviction | Medium |
| Node `MemoryPressure`, `DiskPressure` or `PIDPressure` | Medium |

Node conditions need cluster-wide read access, because nodes sit outside any namespace.

A condition opens an incident only for a workload that is mapped to that namespace and routes the cluster. To route it, open the workload's settings, turn on **Integration alerts** under **Investigation triggers** and select the cluster.

The connector doesn't monitor application health, so for that, add an HTTP health check or connect an alert source. See [Proactive detection](https://tryace.dev/docs/concepts/proactive-detection).

## Before you start

- The **admin** role in Ace.
- A Kubernetes account that can create a namespace, a Secret and RBAC bindings.
- Outbound HTTPS from the cluster to Ace, and access to the connector's image registry (`ghcr.io`, or your mirror).
- Helm 3.8 or later and `kubectl`, or the setup manifest if you don't use Helm.
- 1 to 50 existing namespaces per connection.

### Private networks

The connector works with private clusters, including EKS, GKE and AKS with a private API endpoint, and on-premises clusters. The reader and writer call the Kubernetes API from inside the cluster, so the endpoint doesn't need to be public.

Ace never connects in, so you don't open inbound ports or allowlist Ace. Allow outbound HTTPS from the connector pods in `ace-connectors` to the Ace address in the install commands, and make sure your nodes can pull the image from `ghcr.io` or your mirror.

## Credentials and permissions

Ace generates a credential for each connector, which you download once. The install commands store it as a Secret in the `ace-connectors` namespace.

| Connector | Kubernetes permissions | Why Ace needs them |
| --- | --- | --- |
| Read, in each chosen namespace | Role with `get` and `list` on the resources listed above (no Secrets), plus `pods/log` | Investigations, discovery and incident detection |
| Read, cluster identity | ClusterRole with `get` on the `kube-system` namespace only | Identifies the cluster |
| Read, optional cluster-wide | ClusterRole with `get` and `list` on nodes, namespaces, persistent volumes, storage classes, cluster roles and bindings, and admission webhook configurations | Node conditions and cluster-wide resources |
| Write, in each chosen namespace | Role with `get`, `list`, `patch`, `create` and `delete` on all resources | Approved changes |
| Write, optional **Grant cluster administration** | ClusterRole with all verbs on all resources | Node operations and other cluster-wide changes |

> **Write access reaches data**
>
> Namespace write access can create workloads that read data and credentials in those namespaces. Ace still acts only within a confirmed action scope and your action policy, but review the permissions before you install a writer.

The connector runs as a non-root user with a read-only root filesystem, all Linux capabilities dropped and the `RuntimeDefault` seccomp profile.

## Set up

### Connect the read connector

1. In Ace, go to **Settings** > **Data sources** > **Kubernetes**.
2. Under **Connect a cluster**, enter:
   - **Cluster name**, unique in your workspace, for example `production-eu`.
   - **Namespaces**: comma-separated existing namespaces. Each workload can investigate only its mapped namespaces.
   - **Workload**: the workload to map, or **Map a workload later**.
   - Optional: **Read cluster-wide resources such as nodes and admission webhooks**.
3. Choose **Create connection**.
4. Under **Download the connector credential**, choose **Download credential file**. The file is available only during setup, so keep it until the install finishes.
5. Under **Install with Helm**, enter your **Kubernetes context**:

   ```bash
   kubectl config get-contexts
   ```

6. Choose **Copy Helm commands** and run them from the folder that holds the credential file. They create the `ace-connectors` namespace and the credential Secret, delete the file, install the reader and wait for it to start. The credential never appears on the command line.
7. Under **Check the connection**, choose **Verify connection**. When it shows **Connection verified**, choose **Finish setup**.
8. Under **Workload mapping**, pick a workload and its namespaces, and choose **Save mapping**.

For a mirror or private registry, expand **Use a registry mirror or private registry?** before you copy the commands, then enter the mirrored **Connector image** by digest and, optionally, an **Image pull secret** that already exists in `ace-connectors`.

With Argo CD or Flux, use the same chart and values, and provide the credential Secret through your secret manager.

Without Helm, expand **Install with a manifest instead** and choose **Download setup manifest**. It creates the same objects and also contains the credential, so store it securely.

To connect more clusters, choose **Add another cluster**.

### Allow approved changes (optional)

1. On a verified cluster, choose **Action access**.
2. Optional: turn on **Grant cluster administration**.
3. Choose **Create write connector** and install it like the reader, with the commands Ace generates. It runs as a separate Deployment with its own credential.
4. Under **Workload action scope**, select a mapped workload, its **Environment** and its **Namespaces in this environment**. Each namespace belongs to one environment. Optionally turn on **Permit cluster administration for this workload** or **Permit approved container commands**.
5. Choose **Verify and confirm action scope**.

A writer on its own doesn't turn on automatic repairs.

## Confirm it's working

- The install panel shows **Connection verified** with each check. If it shows **Connection check is pending. Run it again shortly.**, wait for the connector to start and retry.
- If validation times out, check that the connector pod is running, then check DNS, outbound HTTPS and certificate trust from the cluster to Ace. The connector's logs leave out credentials and payloads.
- The cluster card shows the last connection check and its namespaces. **Validate and discover** runs the checks again, and discovery checks **Cluster identity** and **Scoped inventory**.
- If the card asks you to upgrade the connector, follow the steps shown and upgrade the reader and writer together.

## Disconnect

In Ace:

- **Rotate credential** revokes the current credential, so run the install commands again with the new file.
- **Revoke write access** removes the writer's access. **Replace write credential and grant** revokes the old writer credential and invalidates every action scope.
- **Disconnect** deletes the connection.

In the cluster, uninstall the reader and writer with `helm uninstall`, or delete the objects from the setup manifest. Then delete their credential Secrets in `ace-connectors`, using the names from your install commands, and keep the `ace-connectors` namespace if other connectors still run in it.

## Next steps

- [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): approval, automatic policies and undo.
- [Prometheus](https://tryace.dev/docs/integrations/prometheus): add metrics and Alertmanager alerts.
- [Slack](https://tryace.dev/docs/integrations/slack): needed for automatic repair countdowns.