BrokenPipeError when using Attached Service Account on GCE Instance

[dantebarba]

Environment details

• OS type and version: Google Compute Engine (GCE) with Container-Optimized OS
• Python version: Python 3.7.16
• pip version: pip 21.3.1
• google-api-python-client== 1.12.8
• google-auth==1.35.0
• google-auth-httplib2==0.2.0

Steps to reproduce

1. Deploy a Python application on a GCE instance with an attached service account.
2. Use google.auth.default() for authentication.
3. Let the service run for a few hours to allow the token to expire.
4. Perform a Google Drive API call during token refresh (e.g., listing files or creating a folder).

Code example

gdrive_credentials: Credentials = (
    Credentials.from_service_account_info(
        gcp_credentials_dict,
        scopes=SCOPES,
    )
    if gcp_credentials_dict
    else google.auth.default(scopes=SCOPES)[0]
)

def create_services(credentials, scopes):
    drive_service = build(
        "drive", "v3", credentials=credentials
    )
    docs_service = build(
        "docs", "v1", credentials=credentials
    )

    return credentials, drive_service, docs_service

def get_or_create_folder(self, drive_service, folder_name):
    """
    Checks if a folder exists in the root directory and creates it if not.

    :param drive_service: Authorized Drive API client.
    :param folder_name: The name of the folder to check or create.
    :return: The folder ID.
    """
    # Search for the folder in the root directory
    query = f"name = '{folder_name}' and mimeType = 'application/vnd.google-apps.folder' and 'root' in parents and trashed = false"
    response = (
        drive_service.files().list(q=query, fields="files(id, name)").execute()
    )
    files = response.get("files", [])

    if files:
        # Folder exists, return its ID
        return files[0]["id"]
    else:
        # Folder doesn't exist, create it
        folder_metadata = {
            "name": folder_name,
            "mimeType": "application/vnd.google-apps.folder",
            "parents": ["root"],  # Place in the root directory
        }
        folder = (
            drive_service.files()
            .create(body=folder_metadata, fields="id")
            .execute()
        )
        return folder["id"]

Stacktrace

BrokenPipeError: [Errno 32] Broken pipe
  File "/app/endpoints/webhooks.py", line 292, in get_services_status
    current_app.config.get("BUILD_ENVIRONMENT"),
  File "/app/middleware/gdocsprinter.py", line 43, in print_file_with_template
    template_id, gdocs_drive_service, replace_template, build_environment
  File "/app/middleware/gdocsprinter.py", line 111, in create_template_copy
    drive_service, f"{build_environment}-prints"
  File "/app/middleware/gdocsprinter.py", line 86, in get_or_create_folder
    drive_service.files().list(q=query, fields="files(id, name)").execute()
  File "googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "googleapiclient/http.py", line 909, in execute
    headers=self.headers,
  File "googleapiclient/http.py", line 204, in _retry_request
    raise exception
  File "googleapiclient/http.py", line 177, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "google_auth_httplib2.py", line 190, in request
    self._request, method, uri, request_headers)
  File "/usr/local/lib/python3.7/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/usr/local/lib/python3.7/site-packages/google/auth/compute_engine/credentials.py", line 111, in refresh
    self._retrieve_info(request)
  File "/usr/local/lib/python3.7/site-packages/google/auth/compute_engine/credentials.py", line 88, in _retrieve_info
    request, service_account=self._service_account_email
  File "/usr/local/lib/python3.7/site-packages/google/auth/compute_engine/_metadata.py", line 234, in get_service_account_info
    return get(request, path, params={"recursive": "true"})
  File "/usr/local/lib/python3.7/site-packages/google/auth/compute_engine/_metadata.py", line 150, in get
    response = request(url=url, method="GET", headers=_METADATA_HEADERS)
  File "google_auth_httplib2.py", line 117, in __call__
    url, method=method, body=body, headers=headers, **kwargs)
  File "__init__.py", line 1709, in request
    conn, authority, uri, request_uri, method, body, headers, redirections, cachekey,
  File "__init__.py", line 1424, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "__init__.py", line 1347, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "http/client.py", line 1036, in _send_output
    self.send(msg)
  File "http/client.py", line 997, in send
    self.sock.sendall(data)

Brief summary

After switching the authentication method for our application from service account keys to an attached service account on a Google Compute Engine (GCE) instance, we are encountering a BrokenPipeError during API calls to Google Drive.

The issue only occurs during token refresh after the application has been running for a couple of hours. Following the first failure, the service works again without intervention.

Thanks