Powershell never exits on Windows and the build hangs forever

[wilg]

I don't know if it something with my configuration of a local runner, but something is preventing the Docker container from terminating. I think it is that Powershell never exits after completing the build script, even when explicitly exited. I've been taking some stabs in the dark on my fork, but haven't had any success here. https://github.com/wilg/unity-builder

[webbertakken]

Would you mind attaching your workflow file, so we can see your configuration? Could you also tell us what kind of runner you're using?

[wilg]

I'm using self-hosted/local runners, one on Mac, one on Windows. My idea is to have the Mac build Mac IL2CPP and have the Windows machine build Windows IL2CPP and Linux IL2CPP.

The Mac build works properly (though I haven't gotten Steam uploads working yet because of https://github.com/game-ci/steam-deploy/issues/35).

The Windows machine is Windows 11 Pro. I installed Docker via Docker Desktop, maybe that's at issue.

The Windows build fully completes, but the Docker instance hangs. It looks like the Powershell script completes (no code is called after the manual exit I added). However I see that it's possible for Powershell to hang even in this case. https://stackoverflow.com/questions/2041799/powershell-script-gets-stuck-doesnt-exit-when-called-from-batch-file I tried various suggested wacky ways of exiting Powershell, but it didn't work. Maybe I missed one.

I have some puzzling configuration issues with memory with docker on Windows, documented here, but manually setting memory/cpu flags got it working. https://github.com/game-ci/unity-builder/issues/432

If I manually terminate the container from Docker Desktop, docker run exits (though with a crazy status code of what appears to be INT_MAX) and the build proceeds as normal. The player is fully built and uploaded.

name: Build Players
on:
  workflow_dispatch:
    inputs:
      build_mac:
        description: "Build for macOS"
        required: true
        type: boolean
      build_windows:
        description: "Build for Windows"
        required: true
        type: boolean
      build_linux:
        description: "Build for Linux"
        required: true
        type: boolean
env:
  NODE_TLS_REJECT_UNAUTHORIZED: 0
jobs:
  windows_host:
    if: inputs.build_linux || inputs.build_windows
    name: Build for ${{ matrix.platform[1] }} on ${{ matrix.platform[0] }}
    runs-on: [self-hosted, "${{ matrix.platform[0] }}"]
    strategy:
      fail-fast: false
      matrix:
        unityVersion:
          - 2021.3.9f1
        platform:
          - [Windows, StandaloneWindows64]
    steps:
      - uses: actions/checkout@v3
        name: Git Checkout + LFS
        with:
          lfs: true
          fetch-depth: 0
          clean: false

      - uses: wilg/unity-builder@main
        name: Build Unity player
        env:
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
          UNITY_BUILDER_DOCKER_ARGS: "-m 28g --cpus 10 --gpus all"
        with:
          unityVersion: ${{ matrix.unityVersion }}
          targetPlatform: ${{ matrix.platform[1] }}
          buildName: Gumshoe
          allowDirtyBuild: true
          # customParameters: "-myParameter myValue -myBoolean -ThirdParameter andItsValue"

      - uses: actions/upload-artifact@v3
        name: Upload builds artifact
        with:
          name: ${{ matrix.platform[1] }}-build
          path: build/${{ matrix.platform[1] }}
          if-no-files-found: error

  mac_host:
    if: inputs.build_mac
    name: Build for ${{ matrix.platform[1] }} on ${{ matrix.platform[0] }}
    runs-on: [self-hosted, "${{ matrix.platform[0] }}"]
    strategy:
      fail-fast: false
      matrix:
        unityVersion:
          - 2021.3.9f1
        platform:
          - [macOS, StandaloneOSX]
    steps:
      - uses: actions/checkout@v3
        name: Git Checkout + LFS
        with:
          lfs: true
          fetch-depth: 0
          clean: false

      - uses: wilg/unity-builder@main
        id: build
        name: Build Unity player
        env:
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
        with:
          unityVersion: ${{ matrix.unityVersion }}
          targetPlatform: ${{ matrix.platform[1] }}
          buildName: Gumshoe
          allowDirtyBuild: true
          # customParameters: "-myParameter myValue -myBoolean -ThirdParameter andItsValue"

      - uses: wilg/steam-deploy@main
        env:
          STEAM_HOME: ~/Library/Application Support/Steam/
        with:
          username: ${{ secrets.STEAM_USERNAME }}
          password: ${{ secrets.STEAM_PASSWORD }}
          configVdf: ${{ secrets.STEAM_CONFIG_VDF}}
          ssfnFileName: ${{ secrets.STEAM_SSFN_FILE_NAME }}
          ssfnFileContents: ${{ secrets.STEAM_SSFN_FILE_CONTENTS }}
          appId: 2154550
          buildDescription: ${{ steps.build.outputs.buildVersion}}
          rootPath: build
          depot2Path: StandaloneOSX
          releaseBranch: beta

      # - name: Zip build
      #   run: |
      #     ditto -c -k --keepParent --sequesterRsrc build/${{ matrix.platform[1] }}/Gumshoe.app build/${{ matrix.platform[1] }}/artifacts/Gumshoe.app.zip

      # - uses: actions/upload-artifact@v3
      #   name: Upload builds artifact
      #   with:
      #     name: ${{ matrix.platform[1] }}-build-zip
      #     path: build/${{ matrix.platform[1] }}/artifacts
      #     if-no-files-found: error

[wilg]

I had to set allowDirtyBuild: true because the build process seems to create an Editor folder in my assets and copy the build scripts in, which causes uncommitted changes in Git. Not sure how this is intended to work normally.

[webbertakken]

I had to set allowDirtyBuild: true because the build process seems to create an Editor folder in my assets and copy the build scripts in, which causes uncommitted changes in Git. Not sure how this is intended to work normally.

This is added after the check for a dirty branch (or at least supposed to). For linux this works as expected. On windows it should to but I haven't verified this. It is possible for self-hosted runners that don't wipe the workspace to get this error on subsequent runs as well.

[wilg]

Ah, that must be the issue. I don't wipe the workspace (which makes everything super fast!) I guess I can just add it to .gitignore.

[wilg]

Hmmm... I replaced the build script with

Write-Output "Exit 1"

Exit 1

Write-Output "THIS IS AN IMPOSSIBLE LINE"

And it properly exits and continues to the next step...

[wilg]

Narrowed it to

& "c:\steps\build.ps1"

in entrypoint.ps1 causing the hang. so something that happens in the build script is preventing exit. perhaps a process started there somehow never closes

[wilg]

Actually, it's worse. It seems nondeterministic potentially.

[wilg]

I can't say it was quick for me to figure it out, but it looks like removing this line fixes the issue:

# Register the Visual Studio installation so Unity can find it
regsvr32 C:\ProgramData\Microsoft\VisualStudio\Setup\x64\Microsoft.VisualStudio.Setup.Configuration.Native.dll

[webbertakken]

Perhaps it's not being mounted in the docker container. When I tested this locally it worked as expected though.

The exit part is definitely something we need to add, indeed.

[ManuelRauber]

Is there something I can do to help figure out this issue?

I've noticed it when I've switched from 2022.2.5f1 to 2022.2.14f1 (commit: https://github.com/BoundfoxStudios/fairy-tale-defender/commit/79d1cec67909b5d4910d49de59318c9c920ade82) the build for Windows tend to hang so GitHub aborts them after 6 hours.

Example build that hang: https://github.com/BoundfoxStudios/fairy-tale-defender/actions/runs/4811774943 That one was working: https://github.com/BoundfoxStudios/fairy-tale-defender/actions/runs/4811536351

The one that hang just added one line of code: https://github.com/BoundfoxStudios/fairy-tale-defender/commit/88332beb5436dd3c34d6dc3aa769929e75b1d970

So it's not the matter of the code or something.

Also, as @wilg mentioned before it seems totally random.

[davidmfinol]

I've been seeing this issue with my windows builds. I normally just re-run the job after it times out, but this can be pretty annoying. One time I had to retry ~5 times before it finally succeeded.

[szunami]

I've been getting hit with this as well :(

For my builds, I notice that when they hang there isn't output for a long period of time. I wonder if something like inotifywait could be used to kill jobs when this is detected (as a workaround, not a real fix)

Looks like there is a windows port of inotifywait, not sure how mature it is https://github.com/thekid/inotify-win

[msanatan]

I set this up yesterday, at first it worked really well but then it's just been stuck even though the build succeeded. This is the workflow file: https://github.com/msanatan/UnityBuildExample/blob/1934f8fa0c0ddb8a63e7e7cea44d744aaa06f11b/.github/workflows/main.yml

I pretty much took it from their example, stripped it down + the Unity project is just a 3d cube in a screen. The Unity version I'm using is 2021.3.24:

You can see that the build is successful from the logs

Unloading 0 unused Assets / (0 B). Loaded Objects now: 5394.
Memory consumption went from 149.6 MB to 149.6 MB.
Total: 13.652400 ms (FindLiveObjects: 0.792400 ms CreateObjectMapping: 0.592000 ms MarkObjects: 12.247900 ms  DeleteObjects: 0.018500 ms)
###########################
#      Build results      #
###########################
Duration: 00:01:25.2118800
Warnings: 2
Errors: 0
Size: 75665984 bytes
Build succeeded!

But it fails to clean up - https://github.com/msanatan/UnityBuildExample/actions/runs/5032839793/jobs/9026697569. It used to work before, with very minimal changes - https://github.com/msanatan/UnityBuildExample/actions/runs/5024963602/jobs/9022172491

[ManuelRauber]

Getting the same issue now on macOS builds: https://github.com/BoundfoxStudios/fairy-tale-defender/actions/runs/4988496632

[Dids]

Getting the same issue now on macOS builds: https://github.com/BoundfoxStudios/fairy-tale-defender/actions/runs/4988496632

This sounds like a new or different issue, no? I've only seen the OP's issue (or what I believe to be their issue) on Windows builds, running on Windows/Windows Server, and using IL2CPP, where Powershell quite literally hangs and there are no errors or information of any kind that could explain the issue.

EDIT: I do however believe that this could be tied to specific Unity versions, specific versions of Game CI's actions, and specific Unity licenses that seem to be version and OS bound.

[SimonMeysman]

Has anyone found a solution to this yet? More specifically to Powershell not exiting after the build has finished and just timing out? Currently about 40% of my buildruns on github hosted windows machines hang and timeout and have to be restarted manually.

[davidmfinol]

On the discord, someone mentioned this as a workaround:

I changed the exution from & to $process = Start-Process and then I check in a while if $process.HasExited then I continued... and I continously read in the while $process.StandardOutputReadLine()

We'd like to try a similar approach in unity-builder. Accepting PRs for this.

[RedGlow]

It seems I'm having the same problem with a windows local runner, for example I get this log):

Build Succeeded!

###########################
#       Build output      #
###########################



    Directory: C:\github\workspace\build\StandaloneWindows


Mode                LastWriteTime         Length Name                          
----                -------------         ------ ----                          
d-----       10/15/2023  11:47 AM                MonoBleedingEdge              
d-----       10/15/2023  11:47 AM                StandaloneWindows_BurstDebugIn
                                                 formation_DoNotShip           
d-----       10/15/2023  11:47 AM                StandaloneWindows_Data        
-a----       10/15/2023  11:47 AM         650240 StandaloneWindows.exe         
-a----       10/15/2023  11:47 AM         945688 UnityCrashHandler32.exe       
-a----       10/15/2023  11:47 AM       22862360 UnityPlayer.dll               

User *** logged in successfully
[UnityConnect::TryLogin] Request Succeeded.
[Licensing::Module] Trying to connect to existing licensing client channel...
[Licensing::IpcConnector] Connection attempt to the License Client on channel: "LicenseClient-ContainerAdministrator" failed because channel doesn't exist; code: "0x80000002"
[Licensing::Module] Successfully launched the LicensingClient (PId: 2328)
[Licensing::IpcConnector] Successfully connected to the License Client on channel: "LicenseClient-ContainerAdministrator" at "2023-10-15T09:47:32.6063465Z"
[Licensing::Client] Handshaking with LicensingClient:
  Version:                 1.13.1+ca5f299
  Session Id:              ace61a77609d4c5085c6eb5307eb5f79
  Correlation Id:          6c9f5e25c2407ef8f2a2a0bf36157968
  External correlation Id: 255953274004627955
  Machine Id:              tlkLX5L8DGqwDP3eIbRKmwML29M=
[Licensing::Module] Successfully connected to LicensingClient on channel: "LicenseClient-ContainerAdministrator" (connect: 0.67s, validation: 0.04s, handshake: 3.90s)
[Licensing::IpcConnector] Successfully connected to the License Notification on channel: "LicenseClient-ContainerAdministrator-notifications" at "2023-10-15T09:47:36.5420804Z"
[Licensing::Module] Connected to LicensingClient (PId: 2328, launch time: 0.01, total connection time: 4.61s)
[Licensing::Client] Successfully updated the access token
[Licensing::Module] Successfully updated the access token AvjQDMZOX_...
[Licensing::Client] Successfully processed license management request
[Licensing::Module] Successfully returned the entitlement license
[Licensing::Client] Successfully queried for the EntitlementGroupsDetails
[Licensing::Client] Successfully returned ULF license with serial number : "F4-5Z53-U2MK-GFPQ-XSRJ-XXXX"
Exiting without the bug reporter. Application will terminate with return code 0

And then it's stuck until timeout it met.

I've tagged a point in the repo where it surely occours, in case it needs to be forked: https://github.com/owof-games/inkjam23/tree/game-ci-issue-430

PS: I've had to fork @game-ci/unity-builder@v3 because of this bug.

[AndrewKahr]

Try updating to v4 and see if this is solved now

[RedGlow]

Try updating to v4 and see if this is solved now

In my case, v4 solved the problem.