The test runner gets stuck in the "Testing in playmode" phase

[ugnelis]

Bug description

The test runner gets stuck in the "Testing in playmode" phase. I have tried ubuntu-latest and windows-2022 docker images.

CI configuration is almost the same as described in https://game.ci/docs/github/getting-started

name: CI

on:
  workflow_dispatch: { }
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    name: Unity Tests Workflow
    runs-on: ubuntu-latest
    timeout-minutes: 60

    strategy:
      matrix:
        unityVersion:
          - 2020.3.19f1

    steps:

      # Checkout (without LFS)
      - name: Checkout Repository
        uses: actions/checkout@v2

      # Git LFS
      - name: Create LFS file list
        run: git lfs ls-files -l | cut -d' ' -f1 | sort > .lfs-assets-id

      - name: Restore LFS cache
        uses: actions/cache@v2
        id: lfs-cache
        with:
          path: .git/lfs
          key: ${{ runner.os }}-lfs-${{ hashFiles('.lfs-assets-id') }}

      - name: Git LFS Pull
        run: |
          git lfs pull
          git add .
          git reset --hard

      # Cache
      - uses: actions/cache@v2
        with:
          path: Library
          key: Library-${{ hashFiles('Assets/**', 'Packages/**', 'ProjectSettings/**') }}
          restore-keys: |
            Library-

      # Test
      - name: Run Tests
        uses: game-ci/unity-test-runner@v2
        id: tests
        env:
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
        with:
          gitPrivateToken: ${{ secrets.GIT_PRIVATE_TOKEN }}
          githubToken: ${{ secrets.GITHUB_TOKEN }}

      # Test Results
      - name: Upload Test Results
        uses: actions/upload-artifact@v2
        if: always()
        with:
          name: Test Restuls
          path: ${{ steps.tests.outputs.artifactsPath }}

Run Tests logs:

Run game-ci/unity-test-runner@v2
/usr/bin/docker run --workdir /github/workspace --rm --env UNITY_LICENSE --env UNITY_LICENSE_FILE --env UNITY_EMAIL --env UNITY_PASSWORD --env UNITY_SERIAL --env UNITY_VERSION=2020.3.19f1 --env PROJECT_PATH=. --env CUSTOM_PARAMETERS= --env TEST_PLATFORMS=playmode;editmode;COMBINE_RESULTS --env COVERAGE_OPTIONS=generateAdditionalMetrics;generateHtmlReport;generateBadgeReport --env COVERAGE_RESULTS_PATH=CodeCoverage --env ARTIFACTS_PATH=artifacts --env GITHUB_REF --env GITHUB_SHA --env GITHUB_REPOSITORY --env GITHUB_ACTOR --env GITHUB_WORKFLOW --env GITHUB_HEAD_REF --env GITHUB_BASE_REF --env GITHUB_EVENT_NAME --env GITHUB_WORKSPACE=/github/workspace --env GITHUB_ACTION --env GITHUB_EVENT_PATH --env RUNNER_OS --env RUNNER_TOOL_CACHE --env RUNNER_TEMP --env RUNNER_WORKSPACE --env GIT_PRIVATE_TOKEN=*** --env CHOWN_FILES_TO= --volume /home/runner/work/_temp/_github_home:/root:z --volume /home/runner/work/_temp/_github_workflow:/github/workflow:z --volume /home/runner/work/bird-game/bird-game:/github/works...
Unable to find image 'unityci/editor:ubuntu-2020.3.19f1-linux-il2cpp-1' locally
ubuntu-2020.3.19f1-linux-il2cpp-1: Pulling from unityci/editor
08a6abff8943: Pulling fs layer
17bc3eeef474: Pulling fs layer
7525af58113d: Pulling fs layer
643424925c07: Pulling fs layer
a64283161285: Pulling fs layer
6f8bb29b92de: Pulling fs layer
f93bd952912f: Pulling fs layer
2f3943063d01: Pulling fs layer
b94b5611506d: Pulling fs layer
13f2bc070dfd: Pulling fs layer
4f4fb700ef54: Pulling fs layer
7d1b8006d6d1: Pulling fs layer
643424925c07: Waiting
a64283161285: Waiting
6f8bb29b92de: Waiting
f93bd952912f: Waiting
2f3943063d01: Waiting
b94b5611506d: Waiting
13f2bc070dfd: Waiting
4f4fb700ef54: Waiting
7d1b8006d6d1: Waiting
17bc3eeef474: Verifying Checksum
17bc3eeef474: Download complete
08a6abff8943: Verifying Checksum
08a6abff8943: Download complete
643424925c07: Verifying Checksum
643424925c07: Download complete
a64283161285: Verifying Checksum
a64283161285: Download complete
6f8bb29b92de: Verifying Checksum
6f8bb29b92de: Download complete
f93bd952912f: Verifying Checksum
f93bd952912f: Download complete
b94b5611506d: Verifying Checksum
b94b5611506d: Download complete
13f2bc070dfd: Verifying Checksum
13f2bc070dfd: Download complete
4f4fb700ef54: Verifying Checksum
4f4fb700ef54: Download complete
7525af58113d: Verifying Checksum
7525af58113d: Download complete
08a6abff8943: Pull complete
7d1b8006d6d1: Verifying Checksum
7d1b8006d6d1: Download complete
17bc3eeef474: Pull complete
7525af58113d: Pull complete
643424925c07: Pull complete
a64283161285: Pull complete
6f8bb29b92de: Pull complete
f93bd952912f: Pull complete
2f3943063d01: Verifying Checksum
2f3943063d01: Download complete
2f3943063d01: Pull complete
b94b5611506d: Pull complete
13f2bc070dfd: Pull complete
4f4fb700ef54: Pull complete
7d1b8006d6d1: Pull complete
Digest: sha256:d8e7a364522199122b17c750851d5f1e467015b849695188f49e37944520bd49
Status: Downloaded newer image for unityci/editor:ubuntu-2020.3.19f1-linux-il2cpp-1
Changing to "/github/workspace/_activate-license" directory.
/github/workspace/_activate-license /github/workspace
Requesting activation (personal license)
Activation complete.
/github/workspace
/steps/set_gitcredential.sh: line 1: #!/usr/bin/env: No such file or directory
GIT_PRIVATE_TOKEN is set configuring git credentials
---------- git config --list -------------
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
filter.lfs.clean=git-lfs clean -- %f
credential.helper=store
url.https://github.com/.insteadof=ssh://[email protected]/
url.https://github.com/[email protected]
***github.com/.insteadof=https://github.com/
***github.com/.insteadof=ssh://[email protected]/
***github.com/[email protected]:
---------- git config --list --show-origin -------------
file:/etc/gitconfig	filter.lfs.smudge=git-lfs smudge -- %f
file:/etc/gitconfig	filter.lfs.process=git-lfs filter-process
file:/etc/gitconfig	filter.lfs.required=true
file:/etc/gitconfig	filter.lfs.clean=git-lfs clean -- %f
file:/root/.gitconfig	credential.helper=store
file:/root/.gitconfig	url.https://github.com/.insteadof=ssh://[email protected]/
file:/root/.gitconfig	url.https://github.com/[email protected]
file:/root/.gitconfig	***github.com/.insteadof=https://github.com/
file:/root/.gitconfig	***github.com/.insteadof=ssh://[email protected]/
file:/root/.gitconfig	***github.com/[email protected]:
Using project path "/github/workspace/.".
Using artifacts path "artifacts" to save test results.
Using coverage results path "CodeCoverage" to save test coverage results.
Using custom parameters .
Using Unity version "2020.3.19f1" to test.
###########################
#    Artifacts folder     #
###########################
Creating "/github/workspace/artifacts" if it does not exist.
###########################
#    Project directory    #
###########################
total 56K
drwxr-xr-x 9 1001  121 4.0K Jul  1 06:15 .
drwxr-xr-x 4 root root 4.0K Jul  1 06:15 ..
-rw-r--r-- 1 1001  121  217 Jul  1 06:12 .editorconfig
drwxr-xr-x 9 1001  121 4.0K Jul  1 06:12 .git
-rw-r--r-- 1 1001  121 1.9K Jul  1 06:12 .gitattributes
drwxr-xr-x 3 1001  121 4.0K Jul  1 06:12 .github
-rw-r--r-- 1 1001  121 1.3K Jul  1 06:12 .gitignore
drwxr-xr-x 6 1001  121 4.0K Jul  1 06:12 Assets
-rw-r--r-- 1 1001  121  766 Jul  1 06:12 CHARK.BirdGame.Client.csproj.DotSettings
-rw-r--r-- 1 1001  121  766 Jul  1 06:12 CHARK.BirdGame.Editor.csproj.DotSettings
drwxr-xr-x 2 1001  121 4.0K Jul  1 06:12 Packages
drwxr-xr-x 2 1001  121 4.0K Jul  1 06:12 ProjectSettings
drwxr-xr-x 2 root root 4.0K Jul  1 06:15 _activate-license
drwxr-xr-x 2 root root 4.0K Jul  1 06:15 artifacts
###########################
#   Testing in playmode  #
###########################

I used secret var ACTIONS_STEP_DEBUG = true for enabling verbose logs.

Where could be the problem?

[ugnelis]

Probably related https://answers.unity.com/questions/1888121/unity-gets-stuck-on-applicationreload-when-enterin.html

[spaskhalov]

I have messed with same behaviour... Test runner just freezing on "Testing in playmode"

[Peaj]

Same problem here. Locally the tests take around 7s to complete. I canceled the test action after around 20min. Is there any way to get more information on whats going on after "Testing in Playmode" is logged?

[kcarlson]

We're getting the same problem with both edit and playmode tests.

[Andri-gunawan-susanto]

Any update with this problem? I've got the same issue after update unity 2020.3.38f1

[webbertakken]

Any update with this problem?

Updates would be posted in this thread. It doesn't look like a complete failure for projects and all versions, since there's only 5 reports here in 2 to 3 months.

What would help is if everyone added their unity versions, and ideally also some stats on memory usage (you could use atop or other measuring tools).

Either way it may be hard to debug the cause here because the Unity Editor isn't giving us much to work with. Perhaps some kind of debug output would also help. Not sure if anyone managed to get more output than nothing at all?

[kcarlson]

We experienced failures using unity 2019.4.40f1 For now a workaround, we are circumventing the test-runner action and executing something like the following, which works:

- name: Run Tests
  id: tests
  env:
    UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
  run: |
    docker run --gpus all --rm \
        --env UNITY_LICENSE \
        --volume="${PWD}:/opt/project" \
        unityci/editor:ubuntu-2019.4.40f1-linux-il2cpp-1 bash -c /opt/project/ci.sh

ci.sh

#!/bin/bash

FILE_PATH=UnityLicenseFile.ulf
echo "$UNITY_LICENSE" | tr -d '\r' >$FILE_PATH
unity-editor \
    -batchmode \
    -nographics \
    -logFile /dev/stdout \
    -quit \
    -manualLicenseFile $FILE_PATH

platforms=(editmode playmode)

COVERAGE_OPTIONS="pathFilters:+/**/Assets/Scripts/**;assemblyFilters:+assets;generateAdditionalMetrics;generateHtmlReport;generateBadgeReport"

for platform in "${platforms[@]}"; do
    unity-editor \
        -runTests \
        -projectPath /opt/project \
        -testResults /opt/project/Logs/${platform}_test_results.xml \
        -logfile /dev/stdout \
        -debugCodeOptimization \
        -enableCodeCoverage \
        -testPlatform $platform \
        -coverageResultsPath /opt/project/CodeCoverage \
        -coverageOptions $COVERAGE_OPTIONS
done

[Andri-gunawan-susanto]

just found solution in my case, just changing uses: game-ci/unity-test-runner@v2 to uses: game-ci/[email protected] maybe there some run test code in new game-ci version that I can't used in game-ci test runner

[kcarlson]

this solves our issue

[ugnelis]

uses: game-ci/[email protected]

That worked, thanks!

[webbertakken]

Looks like one of the releases caused this. It would be extremely helpful if you could help us find out which one.

Do the versions v2.0.1 and v2.0.2 work for you too?

[ugnelis]

Looks like one of the releases caused this. It would be extremely helpful if you could help us find out which one.

Do the versions v2.0.1 and v2.0.2 work for you too?

I have tested v2.0.1 and v2.0.2. Both of these versions get stuck in the "Testing in playmode" phase

[Peaj]

I am using Unity version 2019.4.28f1.

The project does not have any playmode tests only editmode tests.

[email protected] it hangs at "Testing in Editmode" test-runner@v2 it hangs at "Testing in Playmode" [email protected] it hangs at "Testing in Playmode"

test-runner-log.txt

Trying to run the tests on window-latests also failed:

Run game-ci/unity-test-runner@v2
D:\a\_actions\game-ci\unity-test-runner\v2\dist\main.js:1
index.js
^

ReferenceError: index is not defined
    at Object.<anonymous> (D:\a\_actions\game-ci\unity-test-runner\v2\dist\main.js:1:1)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:17:47

Looks like the simlink is broken?

[dev-cmurphy]

I am using Unity version 2019.4.28f1.

The project does not have any playmode tests only editmode tests.

[email protected] it hangs at "Testing in Editmode" test-runner@v2 it hangs at "Testing in Playmode" [email protected] it hangs at "Testing in Playmode"

test-runner-log.txt

Trying to run the tests on window-latests also failed:

Run game-ci/unity-test-runner@v2
D:\a\_actions\game-ci\unity-test-runner\v2\dist\main.js:1
index.js
^

ReferenceError: index is not defined
    at Object.<anonymous> (D:\a\_actions\game-ci\unity-test-runner\v2\dist\main.js:1:1)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:17:47

Looks like the simlink is broken?

The windows issue is related to this issue it seems.

[oferei]

I also have this problem with play-mode tests. I tried the "2.0.0" solution but it doesn't work for me.

It would also be helpful to flush log messages as they happen instead of dumping them all at the end. Even before this bug the entire bunch of tests log messages would be dumped at one time point--after all the tests were complete--and all having almost the same time stamp.

[webbertakken]

Grouping and flushing output is tracked in https://github.com/game-ci/cli/issues/12

Contributions are welcomed :)

[Peaj]

I just tried running the test on 2021.3.14f1 and instead of getting stuck on "Testing in Playmode" it fails when trying to fetch private git packages. I wonder if that might always be the reason it fails and just does not handle it properly in earlier versions. @ugnelis it seems like you are using private git packages too as there is a "gitPrivateToken" in your config?

[oferei]

I just tried running the test on 2021.3.14f1 and instead of getting stuck on "Testing in Playmode" it fails when trying to fetch private git packages. I wonder if that might always be the reason it fails and just does not handle it properly in earlier versions. @ugnelis it seems like you are using private git packages too as there is a "gitPrivateToken" in your config?

I still have the bug with Unity 2021.3.14f1.

It sounds to me that your fetch failing should not have anything to do with the Unity version.

[ugnelis]

I just tried running the test on 2021.3.14f1 and instead of getting stuck on "Testing in Playmode" it fails when trying to fetch private git packages. I wonder if that might always be the reason it fails and just does not handle it properly in earlier versions. @ugnelis it seems like you are using private git packages too as there is a "gitPrivateToken" in your config?

Probably, you need this step for running test (important line gitPrivateToken):

# Test
- name: Run Tests
  uses: game-ci/unity-test-runner@v2
  id: tests
  env:
    UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
  with:
    gitPrivateToken: ${{ secrets.GIT_PRIVATE_TOKEN }}
    githubToken: ${{ secrets.GITHUB_TOKEN }}

or:

# Build
- uses: game-ci/unity-builder@v2
  env:
    UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
  with:
    gitPrivateToken: ${{ secrets.GIT_PRIVATE_TOKEN }}
    targetPlatform: ${{ matrix.targetPlatform }}
    buildsPath: 'Builds'
    buildName: ${{ needs.setup.outputs.completeName }}

Of course, you need to add Github secret token which has access of private repos (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)

[ugnelis]

I just tried running the test on 2021.3.14f1 and instead of getting stuck on "Testing in Playmode" it fails when trying to fetch private git packages. I wonder if that might always be the reason it fails and just does not handle it properly in earlier versions. @ugnelis it seems like you are using private git packages too as there is a "gitPrivateToken" in your config?

I still have the bug with Unity 2021.3.14f1.

It sounds to me that your fetch failing should not have anything to do with the Unity version.

Tested yesterday game-ci/unity-builder@v2 and game-ci/[email protected], everything works on the Unity version 2020.3.19f1.

[Peaj]

Probably, you need this step for running test (important line gitPrivateToken):

I do have the gitPrivateToken. I am using the same config that works for the builder:

# Test
      - name: Test 🔬
        uses: game-ci/unity-test-runner@v2
        env:
          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
          UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}
        with:
          githubToken: ${{ secrets.GITHUB_TOKEN }}
          gitPrivateToken: ${{ secrets.PRIVATE_TOKEN }}
          customParameters: -username='${{ secrets.USERNAME }}' -password='${{ secrets.PASSWORD }}'

[ugnelis]

Probably, you need this step for running test (important line gitPrivateToken):

I do have the gitPrivateToken. I am using the same config that works for the builder:

# Test
      - name: Test 🔬
        uses: game-ci/unity-test-runner@v2
        env:
          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
          UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}
        with:
          githubToken: ${{ secrets.GITHUB_TOKEN }}
          gitPrivateToken: ${{ secrets.PRIVATE_TOKEN }}
          customParameters: -username='${{ secrets.USERNAME }}' -password='${{ secrets.PASSWORD }}'

If possible, please send here your log file. Also, be sure that you gave enough access for you generated Github access token

[Peaj]

The log just says: Error when executing git command. remote: Invalid username or password For my 3 git hosted packages. But its the same token and user data that work for the build step.

[ugnelis]

Not sure how you are fetching git projects but in our case, we do through UPM as far as I remember. If you are fetching through git directly, this might be useful https://stackoverflow.com/a/73229045/2122457

Just in case I put my current CI configuration that might be useful:

# See: https://game.ci/docs/github/getting-started
name: CI

on:
  workflow_dispatch: { }
  push:
    branches: [ master ]
  pull_request:
    types: [review_requested, ready_for_review]
    branches: [ master ]

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    name: Unity Tests Workflow
    runs-on: ubuntu-latest
    timeout-minutes: 60

    strategy:
      matrix:
        unityVersion:
          - 2020.3.19f1
        testMode:
#          - playmode
          - editmode

    steps:

      # Checkout (without LFS)
      - name: Checkout Repository
        uses: actions/checkout@v2

      # Git LFS
      - name: Create LFS file list
        run: git lfs ls-files -l | cut -d' ' -f1 | sort > .lfs-assets-id

      - name: Restore LFS cache
        uses: actions/cache@v2
        id: lfs-cache
        with:
          path: .git/lfs
          key: ${{ runner.os }}-lfs-${{ hashFiles('.lfs-assets-id') }}

      - name: Git LFS Pull
        run: |
          git lfs pull
          git add .
          git reset --hard

      # Cache
      - uses: actions/cache@v2
        with:
          path: Library
          key: Library-${{ hashFiles('Assets/**', 'Packages/**', 'ProjectSettings/**') }}
          restore-keys: |
            Library-

      # Test
      - name: Run Tests
        uses: game-ci/[email protected]
        id: tests
        env:
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
        with:
          gitPrivateToken: ${{ secrets.GIT_PRIVATE_TOKEN }}
          githubToken: ${{ secrets.GITHUB_TOKEN }}

      # Test Results
      - name: Upload Test Results
        uses: actions/upload-artifact@v2
        if: always()
        with:
          name: Test Restuls
          path: ${{ steps.tests.outputs.artifactsPath }}

If you still experiencing some problems, If I were you, I would try to run Github Actions locally (for example, https://github.com/nektos/act) and try to debug your DevOps scenario on a local machine.

[ugnelis]

I talked with my friend, so in our project, we are not using other private git repos anymore, we have embedded them into Unity project. We have done that as not all team members use Git or have access to all private repos.

In your case, you need to follow game-ci docs on how to set github private repos fetching.

[Peaj]

I talked with my friend, so in our project, we are not using other private git repos anymore, we have embedded them into Unity project. We have done that as not all team members use Git or have access to all private repos.

In your case, you need to follow game-ci docs on how to set github private repos fetching.

Thanks for your help. As mentioned the same settings work for the Build step to get the private packages. Running the actions locally is a good idea. I will try that out. Also to figure out the original issue.

[oferei]

I'd like to return to the error in the OP - the test runner gets stuck. Any info I can add to help with handling this issue?

[ugnelis]

I'd like to return to the error in the OP - the test runner gets stuck. Any info I can add to help with handling this issue?

Have you tried uses: game-ci/[email protected] (putting v2.0.0 instead of v2)?

[oferei]

Yes. Unfortunately it didn't help in my case. (I wrote this above)

[kenorb]

I've got the same hang with game-ci/unity-test-runner@v2. See: https://github.com/game-ci/unity-actions/issues/168 I was able to reproduce it on the empty project with a public repository right after cache was restored. And I'm using Linux runner.

Build: https://github.com/Ecilos/ecilos-unity-assets-sandbox/actions/runs/4288532708/jobs/7470635266#step:4:141

Workflow file: https://github.com/Ecilos/ecilos-unity-assets-sandbox/actions/runs/4288532708/workflow