# Google Cloud

> Connect Google Cloud so Ace can map your projects, read Cloud Monitoring, Logging, audit logs and traces during investigations, open incidents from Cloud Monitoring alerts, scan for savings and, if you allow it, make approved changes.

Source: https://tryace.dev/docs/integrations/gcp

Ace reads Google Cloud to see what is failing, what changed and how your services connect. You connect once, at the organization, a folder or a single project, and setup runs as one command in Cloud Shell or as Terraform. Write access comes from a separate, optional service account, so you can grant it later or never.

| | |
| --- | --- |
| Category | Infrastructure, telemetry and change context |
| Ace reads | Cloud Asset Inventory, Cloud Monitoring metrics and alerts, Cloud Logging, Admin Activity audit logs, Cloud Trace, Error Reporting, Service Health, Recommender findings |
| Ace can change | Nothing with the read account. With the optional actions account, the services you choose, after approval |
| Opens incidents | Yes, from Cloud Monitoring alerts |
| Setup | One command in Cloud Shell, or Terraform. Keyless by default |

## What it reads
During an investigation, Ace reads:

- Cloud Monitoring time series for the incident window, and the alerts open on the affected resources.
- Cloud Logging entries, such as the first errors from a Cloud Run service or a VM.
- Admin Activity audit logs, which show who changed what, and when.
- Cloud Trace spans and Error Reporting groups for the failing service.
- Google Cloud Service Health, so an incident on Google's side is visible next to your own signals.
- Cloud SQL Query Insights for the affected databases.

Ace reads only the resources near the failing workload on the System Map, and every read is bounded and audited.

For the System Map, Ace reads Cloud Asset Inventory across your connected scope when you connect, once a day and when you choose **Sync**: projects, networks and subnets, Cloud Run services and functions, VMs and instance groups, GKE clusters and node pools, load balancers, Cloud SQL, Memorystore, Spanner, Pub/Sub, Cloud Tasks, Cloud Scheduler, Cloud DNS and more. It draws edges from each resource's configuration, such as a load balancer to its backends, a Cloud Run service to its database and secrets, or a Pub/Sub subscription to its push endpoint. AWS and Google Cloud share one map, so a Cloud Run service that calls an endpoint in AWS shows as a cross-cloud dependency.

The read account uses Google's predefined viewer roles. It can't read Secret Manager values, Cloud Storage objects, database rows, queue messages or Data Access audit logs. Three of its roles return more than metadata, and Ace says so beside each one in setup:

- Cloud Asset Inventory returns each resource's configuration, which includes Cloud Run and Cloud Run functions environment variable values and Compute Engine metadata. Ace uses them in memory to find hostnames for the System Map and keeps only the variable names.
- Cloud Logging returns your application logs.
- Cloud Trace's narrowest role can also manage trace scopes. Ace never uses that.

Ace also scans Google's Recommender for savings: idle and oversized VMs and instance groups, idle disks and IP addresses, idle and overprovisioned Cloud SQL instances, Cloud Run cost, and commitments. It checks daily, and findings appear under [**Cost opportunities**](https://app.tryace.dev/settings/integrations/gcp#cost-opportunities) on the Google Cloud page in Ace.

## What it can change
The read account can't create, update or delete anything. To let Ace make changes, add the separate actions account with at least one of these groups:

| Group | Services |
| --- | --- |
| Compute | Cloud Run, Cloud Run functions, instance groups, VMs, GKE node pools |
| Data | Cloud SQL, Memorystore, Spanner, Pub/Sub, Cloud Tasks, Cloud Scheduler |
| Network and edge | Load balancers, Cloud DNS, Cloud Armor, firewall rules |
| Deployments | Cloud Deploy, Workflows |
| Runbooks | Workflows and Cloud Run jobs you allow |
| Alert snoozing | Snooze an alert policy during a fix |

Each group is one custom role that holds only the permissions its actions use. None of the roles can read data, change IAM, act as another identity or get onto a machine through SSH, OS Login, metadata or the serial console.

A change has to clear three checks:

1. The actions account holds the group's role. Ace asks Google which permissions it holds when you turn actions on, when resources are confirmed, and again before every plan and every write.
2. Ace proposes changes only on the Google Cloud resources an admin confirms in each workload's **Cloud resources** settings (`https://app.tryace.dev/workloads/<workload-id>/settings/resources`).
3. A person approves the exact plan, and Ace reads the resource again before it writes. If anything changed since the plan, it doesn't write.

Ace also narrows the riskier operations itself:

- Cloud DNS changes take A, AAAA, CNAME and SRV records only, so Ace can't change mail routing or domain verification records.
- Turning a firewall rule on or off says whether access opens or closes, and for which ranges.
- Cloud Run scaling and restart deploy a new revision, so Ace refuses them while traffic is pinned to a specific revision, such as after a rollback.
- Memorystore failover, removing a Cloud SQL replica and changing a Cloud Armor rule's action always need a person.

Changes to data stores, such as Cloud SQL, Memorystore and Spanner, never run automatically. Every Google Cloud action is listed in the [action catalog](https://tryace.dev/docs/concepts/actions-and-approvals#google-cloud).

Ace changes resources through Google's API. If Terraform, Pulumi or Config Connector manages a resource, its next apply puts the change back. The plan says so when the resource's labels show a tool manages it, or when the workload's mapped repository declares it. It's a caution and never blocks the change, so your team can follow up in code.

Actions need keyless access. A connection that reads with a service account key can't turn actions on, because Ace never makes changes with a stored key.

## What opens an incident

Ace checks Cloud Monitoring alerts every 2 minutes with the read account, so you don't need a webhook or notification channel. With more than 25 projects, each check covers 25, those with open alerts first and the rest in turn.

- An alert that opens on a project in the connection opens an incident or joins an open one.
- The incident resolves when Google closes the alert.

An alert opens an incident only for a workload that routes Google Cloud. To route it, open the workload's settings, go to **Alerts** (`https://app.tryace.dev/workloads/<workload-id>/settings/alerts`), turn on **Investigate alerts** and tick Google Cloud under **Sources**.

Admin Activity audit log entries, checked every 2 minutes, join the change timeline on the resource they changed, whether or not a workload routes Google Cloud, so Ace can line up a config or infrastructure change with an incident. Ace keeps who changed what and when, never a value. Changes Google makes on its own, such as autoscaler resizes and host maintenance, aren't on the timeline, though Ace can still search for them in the logs during an investigation.

## Before you start

You need the **admin** role in Ace (see [Roles and permissions](https://tryace.dev/docs/security/roles-and-permissions)) and a project to hold Ace's service accounts, called the host project. Any project in the organization works, and Ace adds nothing else to it. For a single-project connection, it's that project. A connection covers up to 1,000 projects. [**Manage resources**](https://console.cloud.google.com/cloud-resource-manager) lists every organization, folder and project with its ID, as do `gcloud organizations list` and `gcloud projects list`.

Whoever runs setup in Google Cloud needs these roles:

| Where | Role | For |
| --- | --- | --- |
| Host project | Owner, or Service Usage Admin, Service Account Admin and IAM Workload Identity Pool Admin | Enabling APIs, creating the service accounts and the keyless trust |
| Organization or folder Ace covers | Organization Administrator, or Folder IAM Admin for a folder | Granting the read roles |
| Organization, only with actions | Organization Role Administrator | Creating the custom action roles. A folder's roles live in its organization |

For a single project, Owner on that project covers all of it.

An IAM admin grants these roles in [**IAM**](https://console.cloud.google.com/projectselector/iam-admin/iam?supportedpurview=project,folder,organizationId) for the host project, folder or organization.

Owning an organization doesn't include Organization Role Administrator, so even the person who created it usually needs it granted. The setup script checks every role before it changes anything. If one is missing, it stops with nothing changed and prints the command that grants it, for example:

```bash
gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
  --member user:you@example.com --role roles/iam.organizationRoleAdmin --condition None
```

If you can already set IAM on the organization, the script offers to grant you the role, asks first, and prints the command that removes it afterwards.

If your organization blocks workload identity pools, use a service account key for read-only access.

### Review before you run it

Setup is one public script, the same for every customer: `https://app.tryace.dev/setup/gcp.sh`. Your security team can read it before anyone runs it. To see exactly what it would do in your organization, with every role and permission, add `--plan` to the command Ace shows. It changes nothing:

```bash
curl -fsSL https://app.tryace.dev/setup/gcp.sh | bash -s -- --identity ACE_IDENTITY --actions compute --plan
```

What Ace gets is listed under [Credentials and permissions](#credentials-and-permissions). Neither service account holds Owner, Editor or any other broad role, and the read account can't read Secret Manager values, Cloud Storage objects, database rows, queue messages or Data Access audit logs.

### Split setup between two people

Often the engineer connecting Ace doesn't administer the organization's IAM. Setup runs in two parts:

1. The engineer adds `--part project` to the command and runs it. It creates the service accounts and the keyless trust in the host project, then prints the command for the second part with every value filled in.
2. Someone who administers IAM on the organization or folder runs that command. It grants the read roles and, with actions, creates and grants the action roles. It needs no access to the host project.
3. The engineer pastes the connection code into Ace. Until the second part runs, **Test connection** reports the read roles as missing.

## Credentials and permissions

With keyless access, which is the default, Ace stores no Google credential. Setup creates a workload identity pool in the host project that trusts exactly one identity: the Ace session made for your organization. Ace exchanges that identity for a short-lived token of the read service account. There is no key to leak or rotate.

If your organization can't create a pool, you can connect with a service account key instead. Ace stores the key encrypted and can only read with it.

| Account | Where it lives | What it allows | Why Ace needs it |
| --- | --- | --- | --- |
| `ace-read` | The host project, with roles on the scope you connect | Google's viewer roles for assets, monitoring, logging, trace, error reporting, recommendations, Service Health and service usage | Investigations, the System Map, alerts, the change timeline and cost scans |
| `ace-actions` | The host project, only if you choose action groups | One custom role per group you choose | Approved changes and alert snoozing |

Roles bound on an organization or folder reach every project under it, including projects created later. On connect and on every check, Ace also asks whether the account holds permissions it never uses, and the Google Cloud page in Ace names the broader role they come from so you can remove it.

## Set up

### Connect the read account

1. Open [**Settings** > **Integrations** > **Google Cloud**](https://app.tryace.dev/settings/integrations/gcp).
2. Under **Choose what Ace covers**, choose a **Scope**:
   - **Organization (recommended)** covers every project in the organization.
   - **One folder** covers the projects in a folder. Enter its **Folder ID**, which is a number.
   - **One project** covers one project.
3. Choose **Access**: **Keyless (recommended)** or **Service account key**.
4. Under **Run the setup in Google Cloud**, choose **Open Cloud Shell**. In [Cloud Shell](https://console.cloud.google.com/?cloudshell=true), select the host project in the project picker, then paste and run the one-line command Ace shows. To name another host project, add `--project PROJECT_ID` to the end of the command.
5. The script finds the project and its organization and checks your roles. It shows what it will do and asks before it changes anything. When it finishes, it prints a connection code that starts with `ace-gcp:`.
6. Under **Paste the code it printed**, paste the code into **Connection code**. For a service account key, also paste the contents of `ace-read-key.json` into **Service account key**, then delete the file. Choose **Connect Google Cloud**.

It's safe to run again.

To use Terraform instead, choose **Terraform** under **Setup format**, enter the **Organization ID** or **Project ID** and, for an organization or folder, the **Host project ID**. Apply the Terraform Ace shows with the [roles above](#before-you-start), then paste its `ace_connection_code` output as the connection code. Terraform is available with keyless access only.

Ace checks the account's permissions, then reads your inventory for the System Map.

### Add actions

Actions need keyless access.

1. In [**Settings** > **Integrations** > **Google Cloud**](https://app.tryace.dev/settings/integrations/gcp#actions), find **Actions**.
2. Under **Choose what Ace can change**, select the groups.
3. Choose **Show the setup**. Under **Run the setup in Google Cloud**, run the command in [Cloud Shell](https://console.cloud.google.com/?cloudshell=true) or apply the Terraform Ace shows. Creating the roles needs Organization Role Administrator, as in [Before you start](#before-you-start). It adds the `ace-actions` account and leaves read access as it is.
4. When the setup finishes, choose **Verify and turn on**. Ace checks which groups the account holds and stores no credential.
5. For each workload, go to its settings, open **Cloud resources** (`https://app.tryace.dev/workloads/<workload-id>/settings/resources`) and choose **Google Cloud**. Tick the resources Ace may change and choose **Confirm resources**.

Resources linked to the workload start ticked, and its direct dependencies start unticked. You can search the System Map for others or add one by its full resource name, such as `//run.googleapis.com/projects/my-project/locations/us-central1/services/checkout`. Removing a resource cancels the plans waiting for approval on it, and other plans stay.

## Set up with an agent

An agent can do the Google Cloud side from the terminal. An admin connects it in Ace. See [how agent setup works](https://tryace.dev/docs/integrations#set-up-with-an-agent). The agent runs setup in place of Cloud Shell, with the [gcloud CLI](https://docs.cloud.google.com/sdk/docs/install-sdk) signed in as someone who holds the [roles above](#before-you-start).

1. **Admin: copy the setup command.** Open [**Settings** > **Integrations** > **Google Cloud**](https://app.tryace.dev/settings/integrations/gcp) and choose the **Scope**, **Access** and any groups under **Allow actions (optional)**. Give the agent the command under **Run the setup in Google Cloud**. It carries the one Ace identity Google will trust, so the agent never writes its own.
2. **Agent: run the setup.** Find the IDs, preview the plan, then run it:

   ```bash
   gcloud auth login
   gcloud organizations list
   gcloud projects list
   curl -fsSL https://app.tryace.dev/setup/gcp.sh | bash -s -- <flags from the command> --project <host-project-id> --plan
   curl -fsSL https://app.tryace.dev/setup/gcp.sh | bash -s -- <flags from the command> --project <host-project-id> --yes
   ```

   - The agent runs the `--yes` command only after you approve the plan. An agent's shell usually can't answer the script's prompt, so it needs `--yes`.
   - If a role is missing, the script stops with nothing changed. It prints the `gcloud` command that grants the role, for an IAM admin to run.
   - To [split setup](#split-setup-between-two-people), add `--part project`. The script prints the second command for the IAM admin.
   - With a service account key, the agent moves `ace-read-key.json` to `~/.ace-setup/gcp-read-key`.
3. **Admin: connect Google Cloud.** Under **Paste the code it printed**, enter:
   - **Connection code**: the `ace-gcp:` line the script printed last
   - **Service account key**, for key access only: contents of `~/.ace-setup/gcp-read-key`

   Choose **Connect Google Cloud**, then delete the key file.
4. **Agent: verify.** Check the accounts, the keyless trust and the read account's roles:

   ```bash
   gcloud iam service-accounts describe ace-read@<host-project-id>.iam.gserviceaccount.com --project <host-project-id>
   gcloud iam workload-identity-pools describe ace-ai --location global --project <host-project-id>
   gcloud organizations get-iam-policy <organization-id> \
     --flatten 'bindings[].members' \
     --filter 'bindings.members:serviceAccount:ace-read@<host-project-id>.iam.gserviceaccount.com' \
     --format 'value(bindings.role)'
   ```

   For a folder or project, use `gcloud resource-manager folders get-iam-policy <folder-id>` or `gcloud projects get-iam-policy <project-id>` with the same flags. The last command lists the read account's roles. Then the admin chooses **Test connection**, and **Verify and turn on** under [**Actions**](https://app.tryace.dev/settings/integrations/gcp#actions) if they chose groups.

## Confirm it's working

- The **Connection** section of the [Google Cloud page](https://app.tryace.dev/settings/integrations/gcp) shows the organization, folder or project, and **Keyless access** or **Service account key** with the service account.
- **Test connection** passes. If **Read roles missing** appears, the read roles aren't granted yet. If you [split setup](#split-setup-between-two-people), the second part hasn't run. Otherwise run the setup again, then test the connection.
- **System Map** shows the last sync, and your projects appear on the System Map within a few minutes.
- With actions on, **Actions** shows the actions account with the groups it holds and when it was verified.

## Disconnect

To turn off actions only, choose **Turn off** in [**Actions**](https://app.tryace.dev/settings/integrations/gcp#actions). Ace stops making Google Cloud changes, and the service account and its roles stay in your project until you delete them.

To disconnect Google Cloud, choose **Disconnect** on the Google Cloud page. Ace stops reading your Google Cloud resources, but it doesn't delete anything in Google Cloud. To remove Ace's access:

- Delete the `ace-read` and `ace-actions` [service accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) and the `ace-ai` [workload identity pool](https://console.cloud.google.com/iam-admin/workload-identity-pools) from the host project.
- Remove their role bindings from the organization, folder or project in **IAM**.
- Delete the custom roles named `aceActions*` in the organization's [**Roles**](https://console.cloud.google.com/iam-admin/roles).

To delete the accounts and the pool with gcloud:

```bash
gcloud iam service-accounts delete ace-read@<host-project-id>.iam.gserviceaccount.com --project <host-project-id>
gcloud iam service-accounts delete ace-actions@<host-project-id>.iam.gserviceaccount.com --project <host-project-id>
gcloud iam workload-identity-pools delete ace-ai --location global --project <host-project-id>
```

## Next steps

- [Actions and approvals](https://tryace.dev/docs/concepts/actions-and-approvals): approval, automatic policies and undo.
- [AWS](https://tryace.dev/docs/integrations/aws): connect AWS to the same System Map.
- [Security and data](https://tryace.dev/docs/security/security-and-data): what Ace stores and how it is protected.
