Run Flank in GitHub Action

[vorburger]

I would like to explore if it's possible to run (trigger?) Flank (https://flank.github.io/flank/) in the GitHub Action (which now works fine otherwise),

i.e. ./gradlew :engine:runFlank etc.

as in https://github.com/google/android-fhir/blob/master/kokoro/gcp_ubuntu/kokoro_build.sh

If this works, then I'm not sure this project still needs the Google internal only Kokoro infrastructure? (Thus "fixing" e.g. #2300 or #2297 et al.)

This would, to me, lead to a simplification of this project's (to me, seemingly) "complex" CI infra.

[omarismail94]

runFlank runs the Firebase tests. We need a way to authenticate with Firebase if we run things in GitHub, and the only way I can think of is using a JSON key. Exporting a JSON key is not allowed and Im not sure how difficult it is to get an exception. Why this works in Kokoro is becasue it runs on GCP, which uses IAM roles to authenticate as opposed to JSON keys

[vorburger]

Exporting a JSON key is not allowed and Im not sure how difficult it is to get an exception.

I'll reach out privately because I would like to learn more about this...

[vorburger]

OK, I'm learning more about this, so there are 1-2 problems, here:

1. This projects' Firebase GCP project has Org restrictions which make it impossible to export Service Account credentials as a JSON file (which I would then securely store in a GitHub Secret). There is a process around it; I'll try to find out more about if this may be feasible.

2. Required Firebase Roles aren't really clear to me, yet.

[vorburger]

Note to future self: The following is just a Warning and it already always appears and is no problem:

$ ./gradlew :engine:runFlank

> Task :engine:execFlank
version: v21.11.0
revision: ...
session id: ...

Warning: Failed to make bucket for android-fhir-instrumeted-tests
Cause: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

This is NOT related to and not NOT the root cause of problems such as the following:

ftl.run.exception.PermissionDenied: Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://testing.googleapis.com/v1/testEnvironmentCatalog/ios?projectId=android-fhir-instrumeted-tests
{
  "code": 403,
  "errors": [
    {
      "domain": "global",
      "message": "Not authorized for project android-fhir-instrumeted-tests",
      "reason": "forbidden"
    }
  ],
  "message": "Not authorized for project android-fhir-instrumeted-tests",
  "status": "PERMISSION_DENIED"
}

[vorburger]

The 403 above is fixed by granting the "Editor" role to the SA, as per https://firebase.google.com/docs/test-lab/android/continuous. (Note to future self: It seemed like that it didn't seem to work when I added the "Editor" role to an existing SA, but worked if I created a new one to which I added it at creation time? I'm probably just very confused about the GCP Console; using gcloud CLI next time around may be clearer.)

With this I'm half way there, but still fails at the end:

$ ./gradlew :engine:runFlank

> Task :engine:execFlank
version: v21.11.0
revision: 1bf9b6a8ed1287141b66a75ec66b23813a02f2a8
session id: 9d601213-7b90-4cd5-a1f8-18168dcf3916

Warning: Failed to make bucket for android-fhir-instrumeted-tests
Cause: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
AndroidArgs
    gcloud:
      results-bucket: android-fhir-build-artifacts
      results-dir: engine-d42a277b-3f3b-4cd8-8ddc-98a5303f51ad
      record-video: true
      timeout: 45m
      async: false
      client-details:
      network-profile: null
      results-history-name: null
      # Android gcloud
      app: /home/vorburger/git/github.com/google/android-fhir/demo/build/outputs/apk/androidTest/debug/demo-debug-androidTest.apk
      test: /home/vorburger/git/github.com/google/android-fhir/engine/build/outputs/apk/androidTest/debug/engine-debug-androidTest.apk
      additional-apks:
      auto-google-login: false
      use-orchestrator: true
      directories-to-pull:
        - /sdcard/Download
      grant-permissions: all
      type: null
      other-files:
      scenario-numbers:
      scenario-labels:
      obb-files:
      obb-names:
      performance-metrics: true
      num-uniform-shards: null
      test-runner-class: null
      test-targets:
      robo-directives:
      robo-script: null
      device:
        - model: Nexus6P
          version: 24
          locale: en_US
          orientation: portrait
        - model: MediumPhone.arm
          version: 33
          locale: en_US
          orientation: portrait
      num-flaky-test-attempts: 1
      test-targets-for-shard:
      fail-fast: false
      parameterized-tests: default

    flank:
      max-test-shards: 10
      shard-time: -1
      num-test-runs: 1
      smart-flank-gcs-path: 
      smart-flank-disable-upload: false
      default-test-time: 120.0
      use-average-test-time-for-new-tests: false
      files-to-download:
      test-targets-always-run:
      disable-sharding: false
      project: android-fhir-instrumeted-tests
      local-result-dir: results
      full-junit-result: false
      # Android Flank Yml
      keep-file-path: false
      additional-app-test-apks:
      run-timeout: -1
      legacy-junit-result: false
      ignore-failed-tests: false
      output-style: single
      disable-results-upload: false
      default-class-test-time: 240.0
      disable-usage-statistics: false
      output-report: none
      skip-config-validation: false
      custom-sharding-json: 

RunTests

 Smart Flank cache hit: 0% (0 / 20)
  Shard times: 240s, 240s, 240s, 240s, 240s, 240s, 240s, 240s, 240s, 360s

  Saved 1 shards to /home/vorburger/git/github.com/google/android-fhir/engine/build/fladle/results/engine-d42a277b-3f3b-4cd8-8ddc-98a5303f51ad/android_shards.json
  Uploading [android_shards.json] to https://console.developers.google.com/storage/browser/android-fhir-build-artifacts/engine-d42a277b-3f3b-4cd8-8ddc-98a5303f51ad/...

Error on uploading android_shards.json
Cause: com.google.cloud.storage.StorageException: 403 Forbidden
POST https://storage.googleapis.com/upload/storage/v1/b/android-fhir-build-artifacts/o?projection=full&uploadType=multipart
{
  "error": {
    "code": 403,
    "message": "vorburger-test3@android-fhir-instrumeted-tests.iam.gserviceaccount.com does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist).",
    "errors": [
      {
        "message": "vorburger-test3@android-fhir-instrumeted-tests.iam.gserviceaccount.com does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist).",
        "domain": "global",
        "reason": "forbidden"
      }
    ]
  }
}


Total run duration: 0m  9s
        - Preparation: 0m 4s

> Task :engine:execFlank FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':engine:execFlank'.
> Process 'command '/usr/lib/jvm/java-17-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.

* Get more help at https://help.gradle.org

BUILD FAILED in 11s
8 actionable tasks: 3 executed, 5 up-to-date

This is strange, because the Editor role should have the storage.objects.create permission?

[vorburger]

The (2nd) 403 above is because there are actually 2 separate GCP projects involved here:

android-fhir-instrumeted-tests and android-fhir-build (and the android-fhir-build-artifacts GCS bucket is in the latter)

I'll follow up internally at work to get access sorted out.