GCP Batch: Task Logs: appear in GCP Batch Console, but not retrieved by Funnel yet

[lbeckman314]

Overview

Currently Funnel is not fetching the the System Logs are not being fetch from GCP, resulting in missing Task Logs.

Expected Behavior

TES Task System Logs are kept in sync with the GCP Job Logs (e.g. Example Job).

➜ funnel server run --Compute "gcp-batch" --GCPBatch.Project "tes-batch-integration-test" --GCPBatch.Location "us-central1"

➜ funnel task create examples/gcp-batch.json
<TASK ID>

➜ funnel task get <TASK ID>
# Expect that Task Logs appear hear

[!TIP]

gcloud CLI Example

➜ gcloud config set project tes-batch-integration-test

➜ gcloud batch jobs list --location us-central1 | grep d4tm89hurbu0ugsq7eb0
projects/tes-batch-integration-test/locations/us-central1/jobs/d4tm89hurbu0ugsq7eb0            us-central1  SUCCEEDED

➜ gcloud batch jobs describe d4tm89hurbu0ugsq7eb0 --location us-central1
allocationPolicy:
  labels:
    batch-job-id: d4tm89hurbu0ugsq7eb0
  location:
    allowedLocations:
    - regions/us-central1
    - zones/us-central1-a
    - zones/us-central1-b
    - zones/us-central1-c
    - zones/us-central1-f
  serviceAccount:
    email: [email protected]
createTime: '2025-12-12T00:39:05.783694331Z'
logsPolicy:
  destination: CLOUD_LOGGING
name: projects/tes-batch-integration-test/locations/us-central1/jobs/d4tm89hurbu0ugsq7eb0
status:
  runDuration: 1.491929609s
  state: SUCCEEDED
  statusEvents:
  - description: Job state is set from QUEUED to SCHEDULED for job projects/429372495519/locations/us-central1/jobs/d4tm89hurbu0ugsq7eb0.
    eventTime: '2025-12-12T00:39:10.412614104Z'
    type: STATUS_CHANGED
  - description: Job state is set from SCHEDULED to RUNNING for job projects/429372495519/locations/us-central1/jobs/d4tm89hurbu0ugsq7eb0.
    eventTime: '2025-12-12T00:40:18.331700857Z'
    type: STATUS_CHANGED
  - description: Job state is set from RUNNING to SUCCEEDED for job projects/429372495519/locations/us-central1/jobs/d4tm89hurbu0ugsq7eb0.
    eventTime: '2025-12-12T00:40:19.823630466Z'
    type: STATUS_CHANGED
  taskGroups:
    group0:
      counts:
        SUCCEEDED: '1'
      instances:
      - bootDisk:
          image: projects/batch-custom-image/global/images/batch-cos-stable-official-20251121-01-p00
          sizeGb: '30'
          type: pd-balanced
        machineType: e2-highcpu-2
        provisioningModel: STANDARD
        taskPack: '1'
taskGroups:
- name: projects/429372495519/locations/us-central1/jobs/d4tm89hurbu0ugsq7eb0/taskGroups/group0
  parallelism: '1'
  taskCount: '1'
  taskSpec:
    computeResource:
      cpuMilli: '2000'
      memoryMib: '2000'
    environment: {}
    runnables:
    - container:
        commands:
        - sh
        - -c
        - sha256sum /mnt/disks/tes-batch-integration/README.md | tee /mnt/disks/tes-batch-integration/README.md.sha256
        imageUri: alpine
        volumes:
        - /mnt/disks/tes-batch-integration:/mnt/disks/tes-batch-integration:rw
    volumes:
    - gcs:
        remotePath: tes-batch-integration
      mountPath: /mnt/disks/tes-batch-integration
uid: d4tm89hurbu0ugsq7e-ad70ec7e-120a-428e0
updateTime: '2025-12-12T00:40:19.823630466Z'

Console Example

Current Behavior

System Logs remain empty after Job completes:

➜ funnel server run --Compute "gcp-batch" --GCPBatch.Project "tes-batch-integration-test" --GCPBatch.Location "us-central1"

➜ funnel task create examples/gcp-batch.json
<TASK ID>

➜ funnel task get <TASK ID>
{
   ...
  "logs":  [   <---- Empty Logs ❌
    {
      "metadata":  {
        "gcpbatch_name":  "projects/tes-batch-integration-test/locations/us-central1/jobs/d4tm89hurbu0ugsq7eb0",
        "gcpbatch_uid":  "d4tm89hurbu0ugsq7e-ad70ec7e-120a-428e0"
      }
    }
  ],
  ...
}

Test Steps

1. Start Funnel Server 🌀

➜ funnel server run --Compute "gcp-batch" --GCPBatch.Project "tes-batch-integration-test" --GCPBatch.Location "us-central1"

2. Submit Successful Task ✔️

➜ funnel examples gcp-batch > gcp-batch.json

➜ funnel task create examples/gcp-batch.json
<TASK ID>

2. Fetch Logs 📝

➜ funnel task get <TASK ID>
# Check that SystemLogs match that in GCP Batch