Workflows: update experimental and windows workflows

[jcbrill]

Update the experimental and windows workflow files, add an msvc workflow file, and increase the number of generated files in the ninja iterative speedup test.

Changes:

• Increase the number of generated source files in test/ninja/iterative_speedup.py from 200 to 250. Increasing the workload should reduce the likelihood that the ninja tests are slower.
• Exclude the two tests that are prone to consistent failure in the experimental workflow for macos.
• Move testing skip files from root directory to testing/ci directory and adjust workflow files accordingly.
• Add msvc workflow to run test file with optional environment variables.

MSVC workflow optional environment variables added to the test environment:

• POWERSHELL_TELEMETRY_OPTOUT: 1
• PSDisableModuleAnalysisCacheCleanup: 1
• VCPKG_DISABLE_METRICS: 1
• VSCMD_SKIP_SENDTELEMETRY: 1

The intent is to:

• Disable powershell telemetry and automatic module analysis cache cleanup.
• Disable vcpkg metrics.
• Disable visual studio telemetry.

Contributor Checklist:

• [ ] I have created a new test or updated the unit tests to cover the new/changed functionality.
• [x] I have updated CHANGES.txt and RELEASE.txt (and read the README.rst).
• [ ] I have updated the appropriate documentation

[jcbrill]

The two test failures:

• experimental on macos (two ninja tests that fail consistently)

• appveyor (Visual Studio 2022; Environment: WINPYTHON=Python313) appears to "hang" after final test but before reporting results".

  Failed:

  [00:37:29] 1307/1307 (100.00%) C:\Python313\python.exe test\ZIP\ZIPROOT.py
  [00:37:29] PASSED
  [00:37:29] 
  [00:37:29] Test execution time: 0.9 seconds
  

  Different run that passes:

  [00:33:45] 1303/1307 ( 99.69%) C:\Python311\python.exe test\ZIP\ZIP.py
  [00:33:45] PASSED
  [00:33:45] 
  [00:33:45] Test execution time: 5.2 seconds
  [00:33:45] Total execution time for all tests: 1958.1 seconds
  [00:33:45] 
  [00:33:45] NO RESULT from the following 213 tests:
  

@mwichmann Is there anyway to add a "ci skip test" file only for the experimental tests on macos?

[mwichmann]

Unfortunately, it's not easy to tell what caused a "hang"... it's certainly not the last test reported as completed :-)

It looks like it's the ninja shutdown that didn't finish. If that doesn't have a timer on it, it should. It, of course, is the other thing that fails on the Mac (besides the known control-C problem).

[jcbrill]

The windows hang is unusual.

For the experimental macos tests, these two tests seem to be virtually guaranteed to fail lately:

Failed the following 2 tests:
	test/ninja/ninja_handle_control_c_rebuild.py
	test/ninja/shutdown_scons_daemon.py

Was just wondering if it is possible and makes sense to skip these two only for macos for the time being.

[mwichmann]

For the experimental macos tests, these two tests seem to be virtually guaranteed to fail lately:

Just eliminate the "virtually". They always fail.

Was just wondering if it is possible and makes sense to skip these two only for macos for the time being.

It's possible, of course. I've suggested it, too - would like to see a green result. On the other hand, it's a nag to look into this.

[jcbrill]

It's possible, of course. I've suggested it, too - would like to see a green result. On the other hand, it's a nag to look into this.

Green result achieved!

Test skip file passed only for macos in .github/workflows/experimental_tests.yml.

Added macos_ci_skip.txt in root similar to windows_ci_skip.txt.

[bdbaddog]

It's possible, of course. I've suggested it, too - would like to see a green result. On the other hand, it's a nag to look into this.

Green result achieved!

Test skip file passed only for macos in .github/workflows/experimental_tests.yml.

Added macos_ci_skip.txt in root similar to windows_ci_skip.txt.

Please put the CI changes in a separate PR, and also the exclude lists in the .github dir and not in the root dir?

[mwichmann]

the windows one is in the topdir, I don't know we ever talked about whether that was the right place when it was added.

[jcbrill]

Isn't this whole PR CI related?

There are modifications to two existing workflow files and one new test file.

[bdbaddog]

Isn't this whole PR CI related?

There are modifications to two existing workflow files and one new test file.

Yup. Sorry missed that.

Are we setting up an environment in these changes which is unlikely to represent the common user's? ( all the env changes for windows runners..)

[jcbrill]

Are we setting up an environment in these changes which is unlikely to represent the common user's? ( all the env changes for windows runners..)

Perhaps.

It was only recently that the 2 Powershell and 1 VCPkg variables were added. Prior to the recent powershell/vcpkg PR, the behavior of a user's environment for command-line builds behaved differently than the SCons msvc environment.

The VSCmd skip send telemetry has been around for quite awhile and is configured on every PC and VM configuration here locally.

I don't really see the benefit/utility of having MS telemetry enabled when running the test suite. Telemetry and metrics are beyond SCons control. There does not appear to any upside. The MSVC cache does reduce this burden however.

Two of the added values are specific to the msvc environment. The other two are specific to powershell which is known to be used indirectly by VS in the 2022/2025 runners.

I am unfamiliar with the complete SCons test suite but I would surprised if there were any powershell-specific tests.

I don't when or which VS tools send telemetry.

[jcbrill]

@bdbaddog @mwichmann Where should the files windows_ci_skip.txt and macos_ci_skip.txt be moved?

PSDisableModuleAnalysisCacheCleanup could be removed. The description makes it sound like there is an automatic powershell cache cleanup action to try and reduce the size of the module cache.

The remaining variables (POWERSHELL_TELEMETRY_OPTOUT, VCPKG_DISABLE_METRICS, VSCMD_SKIP_SENDTELEMETRY) seem like a good idea. I could be wrong.

Any thoughts?

[bdbaddog]

Ugh. this is why we try to isolate SCons from shell env variables. In general I think we should be testing with the most common values, but since we respond to some of those values, we may want to do some tests with optional values. I'm guessing the defaults to all the above are to send metrics back to MS.

In theory these shouldn't affect the behavior of our MSVC initialization logic, except that they may be slower?

Does GHA allow setting env for one step? Maybe the bulk of tests run as per your current changes, and one (or a few) run with the default values, only the MSVC test?

[jcbrill]

I'm guessing the defaults to all the above are to send metrics back to MS.

That is my guess as well. The MS way seems to be enable metrics and have user's opt out.

In theory these shouldn't affect the behavior of our MSVC initialization logic, except that they may be slower?

That would be my guess as well.

I'm not sure if these impact the tests that are doing performance comparisons or not though (i.e., doing this is faster/slower than doing that). These kind of tests are really dependent on the OS and environment.

The ninja iterative speedup test in particular fails sporadically on windows.

VSCMD_SKIP_SENDTELEMETRY is known to bypass a call to powershell.exe in the msvc batch files. It was the telemetry powershell.exe call that required the default powershell 5 path to be added to the msvc environment for VS2019 and later.

Since the powershell 5 path was added, the only difference should be whether or not telemetry is setup.

Does GHA allow setting env for one step? Maybe the bulk of tests run as per your current changes, and one (or a few) run with the default values, only the MSVC test?

GHA is new to me. Not sure.

It wouldn't be the end of the world to not set any of the metrics variables.

It would be nice to skip the two ninja tests on macos though.

Perhaps running just the "Full Test Suite on Windows" with all of the optional values in addition to none of the values would be useful either via new workflow file or changing the existing file to run twice.

I'm not sure how to know to assess the impact other than running with and without the additional variables.

[jcbrill]

Changes:

• Optional environment variables removed from experimental workflow.
• Modify windows full test workflow to run twice: 1) without optional environment variables and 2) with optional environment variables.
• Increase the generates source files from 200 to 250 in the ninja iterative speedup test. Increasing the workload should increase the likelihood that using ninja is faster.

Outstanding:

• Should windows_ci_skip.txt and macos_ci_skip.txt be moved to a subdirectory?
• Should windows_ci_skip.txt and macos_ci_skip.txt be renamed? For example exclude_windows.txt and exclude_macos.txt.

Should users/developers running the test suites locally use these files as well? If so, I'm not sure that putting them in the .github folder makes sense.

My preference would be to rename the exclude file names to share a common prefix so that they are grouped together in the topdir or a subfolder.

[jcbrill]

@mwichmann @bdbaddog Any chance one of you can cancel the "Full Test Suite on Windows" workflow?

It has been running for more than 1hr20min. Appears to have hung more than an hour ago. Thanks.

One of the interactive tests failed in appveyor tests.

Grrrr...

[bdbaddog]

@mwichmann @bdbaddog Any chance one of you can cancel the "Full Test Suite on Windows" workflow?

It has been running for more than 1hr20min. Appears to have hung more than an hour ago. Thanks.

One of the interactive tests failed in appveyor tests.

Grrrr...

Killed..

[jcbrill]

No additional work planned.

~~At present, appveyor still running from earlier commit:~~ ~~https://ci.appveyor.com/project/SCons/scons/builds/52217095~~

All tests in https://github.com/SCons/scons/pull/4733/commits/6bc2185eaa7875c4afbe9492decf9b8ba4c9f449 passed.

[jcbrill]

Notes:

• While the interactive tests appear to pass on windows, test/Interactive/configure.py failed once locally. It appears as if the stdout output was not in the order expected by the test. I think I saved the output log.
• I believe there is an pre-existing issue with the runtest_win.yml workflow file (and the "copied" runtest_win_msvc.yml workflow file). I think if there are any failures in the test suite the remaining steps are abandoned and the fail log will not be uploaded. Effectively, the fail log is only uploaded if there are no failures which is not particularly useful. ~I suspect it is due to not having a strategy configuration with a fail-fast: false key/value but I can't be sure.~

Edit: On failure, all remaining steps are skipped unless there is a conditional if: always() or if: failure(). This PR was modified so that failed_tests.log archive is uploaded only on failure for select workflow files.

[mwichmann]

This seems to be promising...

[jcbrill]

Two of the workflow files use python 3.11 while the other four workflow files use python 3.12.

Is this by design?

Python	Workflow
3.11	experimental_tests.yml
3.11	framework_tests.yml
3.12	runtest.yml
3.12	runtest-win.yml
3.12	runtest-win-msvc.yml
3.12	scons-package.yml

Reminder to self: add [skip appveyor] for future commits (if any).

[mwichmann]

Two of the workflow files use python 3.11 while the other four workflow files use python 3.12.

Is this by design? Python Workflow 3.11 experimental_tests.yml 3.11 framework_tests.yml 3.12 runtest.yml 3.12 runtest-win.yml 3.12 runtest-win-msvc.yml 3.12 scons-package.yml

Reminder to self: add [skip appveyor] for future commits (if any).

intentional: kind-of-not-really. I'm trying to spread out the versions a bit without forcing a specific test using every supported Python version, which would be "better", but slower and eat up our free minutes. The AppVeyor run, as a result, does 3.7, 3.9, 3.11 and 3.13 on Windows so oldest, newest, and two others are covered at least there. The github ones have only one restriction - can't use 3.13 for the package build, because of the lxml issue. The two Linux builds driven by runtest.yml should use different versions, but I was too lazy to set up the matrix, I let them be the same except for the Ubuntu version. These things are always eligible for adjustment.

[jcbrill]

I'm trying to spread out the versions a bit without forcing a specific test using every supported Python version, which would be "better", but slower and eat up our free minutes.

The AppVeyor run, as a result, does 3.7, 3.9, 3.11 and 3.13 on Windows so oldest, newest, and two others are covered at least there.

I'm onboard with the philosophy. I do something similar locally (e.g., all versions, oldest+newest versions, same versions as SCons, single version, etc.). Typically only with the msvc/msvs tests but occasionally with the full test suite.

It is unfortunate that the AppVeyor runs take 30+ minutes each for windows. Oldest and newest versions wouldn't be the worst idea if runtime were a concern.

The two Linux builds driven by runtest.yml should use different versions, but I was too lazy to set up the matrix, I let them be the same except for the Ubuntu version. These things are always eligible for adjustment.

Would it make sense to pair the ubuntu version with a python version?

For example:

• ('ubuntu-22.04', '3.7')
• ('ubuntu-24.04', '3.13')

If so, what pairings would be preferred?

[jcbrill]

The two Linux builds driven by runtest.yml should use different versions, but I was too lazy to set up the matrix, I let them be the same except for the Ubuntu version. These things are always eligible for adjustment.

Would it make sense to pair the ubuntu version with a python version?

For example:

• ('ubuntu-22.04', '3.7')
• ('ubuntu-24.04', '3.13')

If so, what pairings would be preferred?

Is this something anyone would like done?

I have an updated/tested workflow file locally that would just need the desired python versions changed.

Note: these are pairings of OS+PY (2 runs) and not full matrix combinations of OS x PY (4 runs).

If not, then no more changes planned.

Edit: Or the workflow file could be updated leaving both python versions the same as they are now to make it easier to change in the future.

[mwichmann]

Would it make sense to pair the ubuntu version with a python version?

For example:

* ('ubuntu-22.04', '3.7')

* ('ubuntu-24.04', '3.13')

If so, what pairings would be preferred?

I'd like to see it - like I said, I was too lazy to set that up. It does not have to be in this PR!

The GH images are preinstalled only with "supported" Python versions, so 22.04/3.9 would be the oldest just depending on Github's setup. Of course, you can get more versions from the setup-python action, but then that's an extra install step (the instruction is the same, but it would need to fetch and install it, rather than just flipping the links which is what happens when you pick a preinstalled one).

[mwichmann]

Oh, and as noted somewhere, workflows/scons-package.yml needs to stay on 3.12 because of our docs and lxml; other actions don't have that restriction.

[jcbrill]

The GH images are preinstalled only with "supported" Python versions, so 22.04/3.9 would be the oldest just depending on Github's setup. Of course, you can get more versions from the setup-python action, but then that's an extra install step (the instruction is the same, but it would need to fetch and install it, rather than just flipping the links which is what happens when you pick a preinstalled one).

Hmm. The way that I modified the file was to parameterize the setup-python action:

      - name: Set up Python ${{ matrix.cfg.py }} ${{ matrix.cfg.os }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.cfg.py }}
          cache: 'pip'

So I guess all supported python versions are available?

[jcbrill]

Full workflow file content for the curious:

# This is a basic workflow to help you get started with Actions

name: SCons Full Test Suite on Linux

# Controls when the workflow will run
on:
  # Triggers the workflow on push or pull request events but only for the master branch
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "runtest"
  runtest:
    strategy:
      fail-fast: false
      matrix:
        cfg: [
          {os: 'ubuntu-22.04', py: '3.7'},
          {os: 'ubuntu-24.04', py: '3.13'},
        ]

    runs-on: ${{ matrix.cfg.os }}

    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v4

      - name: Install Ubuntu packages ${{ matrix.cfg.os }}
        run: |
          sudo apt-get update
          sudo apt-get install libtirpc-dev

      - name: Set up Python ${{ matrix.cfg.py }} ${{ matrix.cfg.os }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.cfg.py }}
          cache: 'pip'

      - name: Install Python ${{ matrix.cfg.py }} dependencies ${{ matrix.cfg.os }}
        run: |
          python -m pip install --progress-bar off --upgrade pip
          python -m pip install --progress-bar off -r requirements-dev.txt
          # sudo apt-get update

      - name: runtest ${{ matrix.cfg.os }} ${{ matrix.cfg.py }}
        run: |
          python runtest.py --all --time --jobs=4

      - name: Archive Failed tests ${{ matrix.cfg.os }} ${{ matrix.cfg.py }}
        uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: ${{ matrix.cfg.os }}-${{ matrix.cfg.py }}-failed-tests
          path: |
            failed_tests.log

[bdbaddog]

Looks good to me, commit the change?

[jcbrill]

Completed.