feat: support for on device (Android emulator / iOS simulator) integration tests in shorebird ci

[ABausG]

Description

For developing home_widget part of quality gates for the project is writing integration test verifying that the plugin's native code does what it is expected todo. Using Emulator/Simulator (note: will use just simulator going forward but meaning both) actions in GitHub Actions is quite annoying to work with. I was able to get the build times down to ~10 mins when they work however the tests have the tendency to be quite flaky and running into timeout issues (capping them at 30mins as I don't want to have the macOS Runners run indefinitely ) Out of these 10mins most time is not actually used for running the tests/building the integration test bundle but rather booting up the simulator. Similarly on a local machine booting up a simulator and running the test is a lot quicker. In this I have at times resorted to running integration tests locally to validate an external PR which I think should be the responsibility of a CI System

Requirements

• [ ] Ability to run integration tests on Simulators for both iOS and Android

Additional Context

Example Android Integration Test Run

Android Integration Test Workflow

  android:
    name: Android Integration Tests
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
      - uses: subosito/flutter-action@v1
        with:
          channel: stable
      - uses: bluefireteam/melos-action@v3
      - name: Enable KVM
        run: |
          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
          sudo udevadm control --reload-rules
          sudo udevadm trigger --name-match=kvm
      - name: Set up JDK 21
        uses: actions/setup-java@v4
        with:
          distribution: 'temurin'
          java-version: '21'
      - name: Run Android Integration Tests
        uses: reactivecircus/android-emulator-runner@v2
        with:
          api-level: 33
          arch: x86_64
          profile: Nexus 6
          sdcard-path-or-size: 100M
          emulator-boot-timeout: 900
          emulator-options: -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim -camera-back none
          script: flutter test integration_test/android_test.dart -d emulator-5554
          working-directory: packages/home_widget/example

Example Test run on iOS Simulator Action

iOS Integration Test Workflow

  ios:
    name: iOS Integration Tests
    strategy:
      matrix:
        device:
          - "iPhone 16"
      fail-fast: false
    runs-on: macos-15
    timeout-minutes: 30
    defaults:
      run:
        working-directory: packages/home_widget/example
    steps:
      - uses: actions/checkout@v2
      - uses: maxim-lobanov/setup-xcode@v1
        with:
          xcode-version: latest-stable
      - uses: subosito/flutter-action@v1
        with:
          channel: stable
      - uses: bluefireteam/melos-action@v3
      - uses: futureware-tech/simulator-action@v1
        id: simulator
        with:
          model: ${{ matrix.device }}
      - name: "Run iOS integration tests"
        run: flutter test integration_test/ios_test.dart -d ${{steps.simulator.outputs.udid}} --ignore-timeouts

[eseidel]

I agree, this would be nice and something we can look into after we support being able to build the app from within CI. No plans to do this before Q2 2026 at the earliest.

[ABausG]

Thanks for the response! I ended up migrating the iOS Integration Tests to Codemagic for now. But potentially move the other analytics checks over to shorbird CI!!