Best practices for mitigating compromised OAuth tokens for Google Cloud CLI

Last reviewed 2024-02-15 UTC

This document describes how to mitigate the impact of an attacker compromising the OAuth tokens that are used by the gcloud CLI.

An attacker can compromise these OAuth tokens if they gain access to an endpoint where a legitimate user account or service account has already authenticated with the gcloud CLI. The attacker can then copy these tokens to another endpoint that they control to make requests that impersonate the legitimate identity. Even after you remove the attacker's access to the compromised endpoint, the attacker can continue to make authenticated API requests using the copied tokens. To help mitigate this risk, you can control access to your systems by using credentials that are short-lived and context-aware.

This document is intended for security teams or cloud architects who are responsible for securing their cloud resources from illegitimate access. This document introduces the available controls that you can use to proactively reduce the impact of compromised gcloud CLI OAuth tokens and remediate your environment after an endpoint has been compromised.

Overview

To understand how this threat works, you must understand how the gcloud CLI stores OAuth 2.0 credentials and how those credentials can be abused if compromised by an attacker.

Types of credentials stored by the gcloud CLI

The gcloud CLI uses OAuth 2.0 access tokens to authenticate requests for Google Cloud APIs. The OAuth flow varies by the credential types used, but generally the access token and other credentials are accessible locally. In each case, the access token expires after 60 minutes, but other credential types might be persistent.

When you authorize the gcloud CLI with a user account, the gcloud CLI initiates a three-legged OAuth consent flow to access Google Cloud APIs on the user's behalf. After the user completes the consent flow, the gcloud CLI receives an access token and a refresh token that allows it to request new access tokens. Under the default settings, the long-lived refresh token persists until its expiration conditions are met.

When you authorize the gcloud CLI with a service account, the gcloud CLI initiates a two-legged OAuth flow to access Google Cloud APIs as the service account identity. After you activate a service account from a private key file, this key is used to periodically request an access token. The long-lived private key is stored in the gcloud CLI configuration and remains valid until you delete the service account key.

When you run the gcloud CLI inside a Google Cloud environment, like Compute Engine or Cloud Shell, the application can automatically find credentials and authenticate as a service account. For example, in Compute Engine, an application like the gcloud CLI can query the metadata server for an access token. Google manages and rotates the private signing key that is used to create the access token, and the long-lived credentials aren't exposed to the application.

When you authenticate with workload identity federation, applications authenticate based on a credential from an external identity provider and receive a federated short-lived access token. For more information on how to store and manage the long-lived credentials used by the external identity provider, see best practices for using workload identity federation.

How an attacker can use compromised OAuth tokens

If an attacker manages to compromise an endpoint, credentials such as OAuth tokens are valuable targets because they let attackers persist or escalate their access.

A developer might have a legitimate need to view their own credentials when writing and debugging code. For example, a developer might need to authenticate for using REST requests to Google Cloud services when working with an unsupported client library. The developer can view the credentials through various methods, including the following:

However, an attacker might use these same techniques after they have compromised an endpoint.

If an attacker compromises an endpoint, such as a developer workstation, the primary threat is that the attacker can run gcloud CLI commands or other code with the legitimate credentials of the authenticated identity. In addition, the attacker might copy the OAuth tokens to another endpoint that they control to persist their access. When this credential theft happens, there is a secondary threat that the attacker can still use the long-lived OAuth tokens to have persistent access even after you remove access to the compromised endpoint.

If the attacker manages to compromise OAuth tokens, they can complete the following actions:

  • An attacker can impersonate the compromised user or service account. API traffic that uses the compromised tokens is logged as if it came from the compromised user or service account, making it difficult to distinguish normal and malicious activity in logs.
  • An attacker can refresh the access token indefinitely using a persistent refresh token associated with a user or a private key associated with a service account.
  • An attacker can bypass authentication with the user's password or 2-step verification because the tokens are granted after the sign-in flow.

Best practices for mitigating risks

Implement the controls described in the following sections to help mitigate the risk of compromised gcloud CLI tokens. If you're following the security best practices that are described in the enterprise foundations blueprint or landing zone design in Google Cloud, you might already have these controls in place.

Set session length for Google Cloud services

To reduce how long an attacker can exploit a compromised token, set the session length for Google Cloud services. By default, the refresh token that the system grants after authentication is a long-lived credential and a gcloud CLI session never requires reauthentication. Change this setting to configure a reauthentication policy with a session length that is between 1 and 24 hours. After the defined session length, the reauthentication policy invalidates the refresh token and forces the user to regularly reauthenticate the gcloud CLI with their password or security key.

The session length for Google Cloud services is a distinct setting from session length for Google services, which controls web sessions for sign-in across Google Workspace services but doesn't control reauthentication for the Google Cloud. If you use Google Workspace services, set the session length for both.

Configure VPC Service Controls

Configure VPC Service Controls across your environment to help ensure that only Google Cloud API traffic that originates within your defined perimeter can access supported resources. The service perimeter limits the usefulness of compromised credentials because the perimeter blocks requests to restricted services that originate from attacker-controlled endpoints that are outside of your environment.

Configure Chrome Enterprise Premium

Configure Chrome Enterprise Premium policies to help secure the Google Cloud console and Google Cloud APIs. Configure a Chrome Enterprise Premium access level and binding to selectively allow attributes that are evaluated on every API request, including IP-based access or certificate-based access for mutual TLS. Requests that use compromised authorization credentials but don't meet the conditions that are defined in your Chrome Enterprise Premium policy are rejected.

Chrome Enterprise Premium is a user-centric control that rejects user API traffic that doesn't meet defined conditions. VPC Service Controls is a resource-centric control that defines the perimeters within which resources can communicate. VPC Service Controls applies to all user identities and service account identities, but Chrome Enterprise Premium applies only to user identities within your organization. When used together, Chrome Enterprise Premium and VPC Service Controls reduce the effectiveness of compromised credentials on an attacker-controlled machine that is outside of your environment.

Enforce 2-step verification for remote server access

If you let developers access Compute Engine resources using SSH, configure OS Login with 2-step verification. This enforces an additional checkpoint where a user must reauthenticate with their password or security key. An attacker with compromised OAuth tokens but no password or security key is blocked by this feature.

Remote Desktop Protocol (RDP) access to Windows instances on Compute Engine doesn't support the OS Login service, so 2-step verification can't be granularly enforced for RDP sessions. When using IAP Desktop or Google Chrome-based RDP plugins, set coarse-grained controls like session length for Google services and 2-step verification settings for the user's web sessions and disable the Allow user to trust the device setting under 2-step verification.

Restrict the use of service account keys

When you use a service account key to authenticate, the key value is stored in the gcloud CLI configuration files, separately from the downloaded key file. An attacker with access to your environment could copy the key from the gcloud CLI configuration or copy the key file from your local filesystem or internal code repository. Therefore, in addition to your plan to mitigate compromised access tokens, consider how you manage downloaded service account key files.

Review more secure alternatives for authentication to reduce or eliminate your use cases that depend on a service account key, and enforce the organization policy constraint iam.disableServiceAccountKeyCreation to disable service account key creation.

Consider the principle of least privilege

When designing IAM policies, consider least privilege. Grant users the roles that they require to accomplish a task at the smallest scope. Don't grant them roles that they don't require. Review and apply role recommendations to avoid IAM policies with unused and excessive roles in your environment.

Protect your endpoints

Consider how an attacker might gain physical access or remote access to your endpoints, like developer workstations or Compute Engine instances. While a plan to address the threat of compromised OAuth tokens is important, also consider how to respond to the threat of how an attacker can compromise your trusted endpoints in the first place. If an attacker has access to your trusted endpoints, they can run gcloud CLI commands or other code directly on the endpoint itself.

Although comprehensive protection for developer workstations is beyond the scope of this document, evaluate how your security tools and operations can help protect and monitor your endpoints for compromise. Consider the following questions:

  • How is the physical security of developer workstations protected?
  • How do you identify and respond to network breaches?
  • How do users get remote access to SSH or RDP sessions?
  • How might other persistent credentials like SSH keys or service account keys be compromised?
  • Are there workflows that use persistent credentials that could be replaced with short-lived credentials?
  • Are there shared devices where someone could read another user's cached gcloud CLI credentials?
  • Can a user authenticate with gcloud CLI from an untrusted device?
  • How does approved traffic connect to resources inside your VPC Service Control perimeter?

Ensure that your security operations address each of these questions.

Align your response teams

Ensure in advance that security teams who are responsible for incident response have appropriate access across the Google Cloud console and the Admin Console. If separate teams manage the Google Cloud console and the Admin Console, you might have a delayed response during an incident.

To remove access from a compromised user account, your incident response team needs an Admin Console role, such as User Management Admin. To assess whether suspicious activity has occurred in your Google Cloud resources, this team might also need IAM roles, such as Security Reviewer across all projects or Logs Viewer on a centralized log sink. The necessary roles for your security team will vary based on the design and operation of your environment.

Best practices for remediation after a security incident

After an endpoint is compromised, as part of your incident management plan, determine how to respond to the primary threat of a compromised endpoint and how to mitigate potential ongoing damage from the secondary threat of compromised tokens. If an attacker has persistent access to the developer workstation, they might copy tokens again after the legitimate user reauthenticates. If you suspect that gcloud CLI tokens might be compromised, open a ticket with Cloud Customer Care and complete the recommendations in the following sections. These actions can help limit the impact of an event like this in your Google Cloud organization.

The recommendations in this section overlap with the general guidance on handling compromised Google Cloud credentials, but focus specifically on the threat of gcloud CLI tokens that are copied from a compromised endpoint.

Expire active tokens for all user accounts with Google Cloud session control

If you haven't already enforced Google Cloud session control, immediately enable this with a short reauthentication frequency. This control helps ensure that all refresh tokens expire at the end of the duration that you define, which limits the duration that an attacker can use the compromised tokens.

Manually invalidate tokens for compromised user accounts

Review the guidance for handling compromised credentials for any user identities who could have been compromised. In particular, removing gcloud CLI credentials is the most effective method for a security team to address compromised OAuth tokens for user identities. To immediately invalidate refresh tokens and access tokens for the gcloud CLI and force the user to reauthenticate with their password or security key, remove the gcloud CLI from a user's list of connected applications.

An individual user can also remove gcloud CLI credentials for their individual account.

Other methods, like suspending the user, resetting the user's password, or resetting sign-in cookies don't specifically address the threat of compromised OAuth tokens. These methods are generally useful for incident response but don't invalidate the access tokens the attacker already controls. For example, if you chose to suspend a user during an investigation but don't revoke gcloud CLI tokens, the access tokens might still be valid if the suspended user is restored before the access tokens expire.

Programmatically invalidate tokens for many user accounts

If you suspect a breach but can't identify which users were affected, consider revoking active sessions for all users in your organization more quickly than the reauthentication policy allows.

This approach can be disruptive to legitimate users and terminate long-running processes that depend on user credentials. If you choose to adopt this approach, prepare a scripted solution for your security operations center (SOC) to run in advance and test it with a few users.

The following sample code uses the Workspace Admin SDK to identify all user identities in your Google Workspace or Cloud Identity account who have access to the gcloud CLI. If a user has authorized the gcloud CLI, the script revokes the refresh token and access token and forces them to reauthenticate with their password or security key. For instructions on how to enable the Admin SDK API and run this code, see the Google Apps Script Quickstart.

/**
 * Remove access to the Google Cloud CLI for all users in an organization
 * @see https://developers.google.com/admin-sdk/directory/reference/rest/v1/tokens
 * @see https://developers.google.com/admin-sdk/directory/reference/rest/v1/users
 * @see https://developers.google.com/apps-script/guides/services/advanced#enabling_advanced_services
 */

function listUsersAndInvalidate() {
  const users = AdminDirectory.Users.list({
    customer: 'my_customer' // alias to represent your account's customerId
    }).users;
  if (!users || users.length === 0) {
    Logger.log('No users found.');
    return;
  }
  for (const user of users){
    let tokens = AdminDirectory.Tokens.list(user.primaryEmail).items
    if (!tokens || tokens.length === 0) {
      continue;
    }
    for (const token of tokens) {
      if (token.clientId === "32555940559.apps.googleusercontent.com") {
        AdminDirectory.Tokens.remove(user.primaryEmail, token.clientId)
        Logger.log('Invalidated the tokens granted to gcloud for user %s', user.primaryEmail)
      }
    }
  }
}

Invalidate and rotate credentials for service accounts

Unlike access tokens that are granted to user identities, access tokens that are granted to service accounts can't be invalidated through the Admin Console or commands like gcloud auth revoke. Additionally, the session duration that you specify in Google Cloud session control applies to user accounts in your Cloud Identity or Google Workspace directory, but not to service accounts. Therefore, your incident response for compromised service accounts needs to address both the persistent key files and the short-lived access tokens.

If you suspect credentials for a service account were compromised, disable the service account, delete service account keys if any exist, and then, after 60 minutes, enable the service account. Deleting a service account key can invalidate the long-lived credential so that an attacker can't request a new access token, but it doesn't invalidate the access tokens already granted. To ensure access tokens aren't abused within their 60 minute lifetime, you must disable the service account for 60 minutes.

Alternatively, you can delete and replace the service account to immediately revoke all short-lived and long-lived credentials, but this might require more disruptive work to replace the service account in applications.

What's next