Kubernetes

Ace reads pods, rollouts, events and logs from your cluster during an investigation. The connector runs as Deployments in your cluster, with a read-only reader and an optional, separate writer for approved changes.

Both connect out to Ace over HTTPS, so you don't open an inbound port, expose the Kubernetes API or upload a kubeconfig, and Kubernetes credentials stay in the cluster.

What Ace readsCopy Anchor Link

In each namespace you choose, the reader can get and list these resources and read current and previous container logs:

• Pods, Services, Endpoints, EndpointSlices, ConfigMaps, Events, PersistentVolumeClaims, ResourceQuotas, LimitRanges, ServiceAccounts
• Deployments, ReplicaSets, StatefulSets, DaemonSets, ControllerRevisions, Jobs, CronJobs
• Ingresses, NetworkPolicies, HorizontalPodAutoscalers, PodDisruptionBudgets, Roles, RoleBindings
• Argo CD Applications, Argo Rollouts, Flux Kustomizations and HelmReleases, when your cluster serves them

The reader never reads or writes Secrets.

Read cluster-wide resources such as nodes and admission webhooks adds Nodes, Namespaces, PersistentVolumes, StorageClasses, ClusterRoles, ClusterRoleBindings, and mutating and validating webhook configurations. Without it, the only thing the reader reads outside your chosen namespaces is the kube-system namespace, which it uses to identify the cluster. An investigation reads only the namespaces mapped to the incident's workload.

What Ace can changeCopy Anchor Link

The reader can't change anything. For changes, install a separate write connector and confirm which namespaces it can act on for each workload and environment, and Ace can then propose:

• Rollouts: restart, undo, pause and resume, and set a container image.
• Capacity: scale, update an autoscaler, set container resources, update a disruption budget.
• Jobs: suspend a CronJob. Resume a CronJob or re-run a Job (approval only).
• Configuration: set a non-secret environment value (approval only).
• Pods and nodes: delete a pod, and cordon, uncordon, taint, untaint or drain a node. These are all approval only, and node operations need cluster administration.
• GitOps controllers: Argo Rollouts abort, undo, promote and retry, Argo CD rollback, sync and auto-sync on or off, and Flux reconcile, suspend and resume.
• An exact patch, create or delete on a supported resource.

A person approves each exact operation unless an admin grants automatic repairs, and approval-only operations never run automatically.

Most operations have an Undo, including scale, rollout undo, and image and resource changes. A rollout restart, pod delete, Job re-run and node drain can't be undone, and undoing a drain only uncordons the node.

Automatic repairs are how Earned autonomy works in Kubernetes, and they need their own grant. An admin names the exact resource, field and allowed value, sets an expiry of at most seven days, a daily limit and a recovery window, and then confirms I authorize automatic restoration of these fields and values for this workload and environment.

Ace repairs automatically only within that grant and only after it delivers a countdown to Slack, so your team sees the repair coming. If a repair makes things worse, automatic repairs stop. See Actions and approvals.

What opens an incidentCopy Anchor Link

The reader watches pod, rollout and node state in the mapped namespaces:

Node conditions need cluster-wide read access, because nodes sit outside any namespace.

A condition opens an incident only for a workload that is mapped to that namespace and routes the cluster. To route it, open the workload's settings, turn on Integration alerts under Investigation triggers and select the cluster.

The connector doesn't monitor application health, so for that, add an HTTP health check or connect an alert source. See Proactive detection.

Before you startCopy Anchor Link

• The admin role in Ace.
• A Kubernetes account that can create a namespace, a Secret and RBAC bindings.
• Outbound HTTPS from the cluster to Ace, and access to the connector's image registry (ghcr.io, or your mirror).
• Helm 3.8 or later and kubectl, or the setup manifest if you don't use Helm.
• 1 to 50 existing namespaces per connection.

Private networksCopy Anchor Link

The connector works with private clusters, including EKS, GKE and AKS with a private API endpoint, and on-premises clusters. The reader and writer call the Kubernetes API from inside the cluster, so the endpoint doesn't need to be public.

Ace never connects in, so you don't open inbound ports or allowlist Ace. Allow outbound HTTPS from the connector pods in ace-connectors to the Ace address in the install commands, and make sure your nodes can pull the image from ghcr.io or your mirror.

Credentials and permissionsCopy Anchor Link

Ace generates a credential for each connector, which you download once. The install commands store it as a Secret in the ace-connectors namespace.

The connector runs as a non-root user with a read-only root filesystem, all Linux capabilities dropped and the RuntimeDefault seccomp profile.

Set upCopy Anchor Link

Connect the read connectorCopy Anchor Link

1. In Ace, go to Settings > Data sources > Kubernetes.

2. Under Connect a cluster, enter:

   • Cluster name, unique in your workspace, for example production-eu.
   • Namespaces: comma-separated existing namespaces. Each workload can investigate only its mapped namespaces.
   • Workload: the workload to map, or Map a workload later.
   • Optional: Read cluster-wide resources such as nodes and admission webhooks.

3. Choose Create connection.

4. Under Download the connector credential, choose Download credential file. The file is available only during setup, so keep it until the install finishes.

5. Under Install with Helm, enter your Kubernetes context:

   Copy Text

   kubectl config get-contexts

6. Choose Copy Helm commands and run them from the folder that holds the credential file. They create the ace-connectors namespace and the credential Secret, delete the file, install the reader and wait for it to start. The credential never appears on the command line.

7. Under Check the connection, choose Verify connection. When it shows Connection verified, choose Finish setup.

8. Under Workload mapping, pick a workload and its namespaces, and choose Save mapping.

For a mirror or private registry, expand Use a registry mirror or private registry? before you copy the commands, then enter the mirrored Connector image by digest and, optionally, an Image pull secret that already exists in ace-connectors.

With Argo CD or Flux, use the same chart and values, and provide the credential Secret through your secret manager.

Without Helm, expand Install with a manifest instead and choose Download setup manifest. It creates the same objects and also contains the credential, so store it securely.

To connect more clusters, choose Add another cluster.

Allow approved changes (optional)Copy Anchor Link

1. On a verified cluster, choose Action access.
2. Optional: turn on Grant cluster administration.
3. Choose Create write connector and install it like the reader, with the commands Ace generates. It runs as a separate Deployment with its own credential.
4. Under Workload action scope, select a mapped workload, its Environment and its Namespaces in this environment. Each namespace belongs to one environment. Optionally turn on Permit cluster administration for this workload or Permit approved container commands.
5. Choose Verify and confirm action scope.

A writer on its own doesn't turn on automatic repairs.

Confirm it's workingCopy Anchor Link

• The install panel shows Connection verified with each check. If it shows Connection check is pending. Run it again shortly., wait for the connector to start and retry.
• If validation times out, check that the connector pod is running, then check DNS, outbound HTTPS and certificate trust from the cluster to Ace. The connector's logs leave out credentials and payloads.
• The cluster card shows the last connection check and its namespaces. Validate and discover runs the checks again, and discovery checks Cluster identity and Scoped inventory.
• If the card asks you to upgrade the connector, follow the steps shown and upgrade the reader and writer together.

DisconnectCopy Anchor Link

In Ace:

• Rotate credential revokes the current credential, so run the install commands again with the new file.
• Revoke write access removes the writer's access. Replace write credential and grant revokes the old writer credential and invalidates every action scope.
• Disconnect deletes the connection.

In the cluster, uninstall the reader and writer with helm uninstall, or delete the objects from the setup manifest. Then delete their credential Secrets in ace-connectors, using the names from your install commands, and keep the ace-connectors namespace if other connectors still run in it.

Next stepsCopy Anchor Link

• Actions and approvals: approval, automatic policies and undo.
• Prometheus: add metrics and Alertmanager alerts.
• Slack: needed for automatic repair countdowns.