Add support for building for Windows ARM64 devices

[anthony-linaro]

Description of Change(s)

Add support for Windows ARM64 devices.

This was tested using the following command line:

python build_scripts/build_usd.py --no-python --onetbb --tests --no-imaging C:/WoA/OpenUSD/build_dir

Python is currently switched off due to lack of support for PySide on these devices. I am looking at that seperately.

I also had to disable imaging, as building boost appears to be broken when using newer CMake versions (similar issue to #3285)

TBB is set to OneTBB, as 2020u3 doesn't support these devices.

I also had to make a small modification in testWrapper.py, and change line 134 to pathPattern = pathPattern + r'(\S*)', as otherwise the escape sequence was invalid.

Checklist

[x] I have created this PR based on the dev branch

[x] I have followed the coding conventions

~~[ ] I have added unit tests that exercise this functionality (Reference: testing guidelines)~~

[x] I have verified that all unit tests pass with the proposed changes

[x] I have submitted a signed Contributor License Agreement (Reference: Contributor License Agreement instructions)

[jesschimein]

Filed as internal issue #USD-10450

:heavy_exclamation_mark: Please make sure that a signed CLA has been submitted!

[jesschimein]

/AzurePipelines run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[anthony-linaro]

CLA sorted - approved by Tim Palmer

[pmolodo]

Any reason why the testWrapper.py change you mentioned isn't included here? Otherwise, lgtm

[anthony-linaro]

I figured it was more related to the version of python I have installed (3.12.5), rather than the one used for validation on other devices - as I didn't have those devices/versions to hand, and knew the script already worked as-was on them, I opted not to change it

[asluk]

"Python is currently switched off due to lack of support for PySide on these devices. I am looking at that seperately."

Hi @anthony-linaro , per our call yesterday, you also mentioned that the python bindings are failing to compile -- can you post the error messages you are seeing there?

"I also had to disable imaging, as building boost appears to be broken... "

@sunyab , is it expected that the command line

python build_scripts/build_usd.py --no-python --onetbb --tests C:/WoA/OpenUSD/build_dir

would attempt to build boost in v24.11+ ?

Thanks all!

[anthony-linaro]

Hi @asluk - I have tried to reproduce the compilation errors, but am unable, so will chalk it down to an env issue - with things set up properly, I have most of the tests passing.

However, when I run ctest, I get this:

The following tests FAILED:
          2 - python_properties (Failed)
         47 - python_pickle1 (Failed)
         50 - python_pickle4 (Failed)
        225 - testPlug (Failed)
        666 - testUsdUtilsCreateNewUsdzPackageEditInPlace (Failed)
        708 - testUsdChecker1 (Failed)
        709 - testUsdChecker2 (Failed)
        710 - testUsdChecker3 (Failed)
        711 - testUsdChecker4 (Failed)
        712 - testUsdChecker5 (Failed)
        720 - testUsdChecker13 (Failed)
        778 - testUsdResolverExample (Failed)

I have attached the output file from the tests to this comment. Are these tests expected to fail? If not, do you have any idea why they might be? All env paths are set as instructed after build (bin and lib on path, python on pythonpath)

LastTest.log

[asluk]

@anthony-linaro I don't think those test failures are expected -- can you try rebasing your branch to be on top of the "dev" branch? That is the most up to date commit. Thanks and happy new year!

[sunyab]

@sunyab , is it expected that the command line

python build_scripts/build_usd.py --no-python --onetbb --tests C:/WoA/OpenUSD/build_dir

would attempt to build boost in v24.11+ ?

(sorry for the late reply, was out on vacation)

Yes, I would expect that to build boost in v24.11+. When imaging is enabled the --tests option requires the idiff tool from OpenImageIO for image diffs, and the version of OIIO the build script uses still requires boost.

[sunyab]

However, when I run ctest, I get this:

The following tests FAILED:
          2 - python_properties (Failed)
         47 - python_pickle1 (Failed)
         50 - python_pickle4 (Failed)
        225 - testPlug (Failed)
        666 - testUsdUtilsCreateNewUsdzPackageEditInPlace (Failed)
        708 - testUsdChecker1 (Failed)
        709 - testUsdChecker2 (Failed)
        710 - testUsdChecker3 (Failed)
        711 - testUsdChecker4 (Failed)
        712 - testUsdChecker5 (Failed)
        720 - testUsdChecker13 (Failed)
        778 - testUsdResolverExample (Failed)

Hi @anthony-linaro, the python_* test failures are due to #3384, which is fixed in the dev branch. The other failures don't look familiar to me, so I'm not sure what's going on there.

[anthony-linaro]

Hi both, thanks for the responses!

@sunyab You're right, those python issues are now fixed, thanks to the suggestion from @asluk to merge dev.

Sadly, the list of errors has now changed, and I now get this:

The following tests FAILED:
        225 - testPlug (Failed)
        656 - testUsdUtilsCreateNewUsdzPackageEditInPlace (Failed)
        764 - testUsdChecker1 (Failed)
        765 - testUsdChecker2 (Failed)
        766 - testUsdChecker3 (Failed)
        767 - testUsdChecker4 (Failed)
        768 - testUsdChecker5 (Failed)
        776 - testUsdChecker13 (Failed)
        777 - testUsdChecker14 (Failed)
        778 - testUsdChecker15 (Failed)
        779 - testUsdChecker16 (Failed)
        780 - testUsdChecker17 (Failed)
        781 - testUsdChecker18 (Failed)
        788 - testUsdChecker26 (Failed)
        789 - testUsdChecker27 (Failed)
        799 - testUsdChecker37 (Failed)
        802 - testUsdResolverExample (Failed)

I have once again attached the logs: LastTest.log

Any suggestions of where I could start to look? It looks like it's not seeing the "Tf" python module. The path variable is correct, and added as specified when building (ie, modified PATH and PYTHONPATH) - is there another one that got missed somewhere? Was there an extra step required between build_usd.py and ctest for python to be able to see the module?

[jesschimein]

/AzurePipelines run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[anthony-linaro]

I didn't add support for ARM64EC, as I didn't have a use-case for it - if you guys have one internally I'm unaware of, I can try and add support.

The biggest issue with ARM64EC will likely be around the use of any intrinsics - as it defines both _M_ARM64EC, and _M_X64, use of intrinsics that could be considered "safely" gated behind _M_X64 (or similar) are suddenly not so safe, so I'd need to go over the source with a fine tooth comb. Not impossible, but requires substantially more work than a plain old ARM64 enablement.

[dgovil]

That's fair. At the very least, I'd suggest then adding a caveat to the README somewhere to document that EC isn't supported yet?

[anthony-linaro]

Good idea - I have updated the README.md file to reflect the work done on this PR

[jesschimein]

/AzurePipelines run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[asluk]

Hi both, thanks for the responses!

@sunyab You're right, those python issues are now fixed, thanks to the suggestion from @asluk to merge dev.

Sadly, the list of errors has now changed, and I now get this:

The following tests FAILED:
        225 - testPlug (Failed)
        656 - testUsdUtilsCreateNewUsdzPackageEditInPlace (Failed)
        764 - testUsdChecker1 (Failed)
        765 - testUsdChecker2 (Failed)
        766 - testUsdChecker3 (Failed)
        767 - testUsdChecker4 (Failed)
        768 - testUsdChecker5 (Failed)
        776 - testUsdChecker13 (Failed)
        777 - testUsdChecker14 (Failed)
        778 - testUsdChecker15 (Failed)
        779 - testUsdChecker16 (Failed)
        780 - testUsdChecker17 (Failed)
        781 - testUsdChecker18 (Failed)
        788 - testUsdChecker26 (Failed)
        789 - testUsdChecker27 (Failed)
        799 - testUsdChecker37 (Failed)
        802 - testUsdResolverExample (Failed)

I have once again attached the logs: LastTest.log

Any suggestions of where I could start to look? It looks like it's not seeing the "Tf" python module. The path variable is correct, and added as specified when building (ie, modified PATH and PYTHONPATH) - is there another one that got missed somewhere? Was there an extra step required between build_usd.py and ctest for python to be able to see the module?

I think the testUsdResolverExample failure is expected, and/or might need the example resolver to be bootstrapped in PATH and PYTHONPATH as well (@sunyab or @pmolodo , do you remember the incantation for that?)

@tallytalwar , looks like the usdchecker rules are failing (also affects testUsdUtilsCreateNewUsdzPackageEditInPlace), but I'm not sure why-- does anything in the log stand out to you or have a clear root cause in your experience? (e.g., Shader </World/simpleMaterial/boardMat/PBRShader> has invalid shader node. (fails 'ShaderPropertyTypeConformanceChecker'))

It's not clear to me what the testPlug failure is.

[tallytalwar]

@asluk all the usdchecker tests (both old <13 and new framework >=13) are failing because its not able to find any of the usdShaders, example UsdPreviewSurface etc in sdr:

New compliance checker failing here: https://github.com/PixarAnimationStudios/OpenUSD/blob/dev/pxr/usdValidation/usdShadeValidators/validators.cpp#L341-L353 Old compliance checerk failing here: https://github.com/PixarAnimationStudios/OpenUSD/blob/dev/pxr/usd/usdUtils/complianceChecker.py#L712

Both these point to a build issue which is not picking up some plugins example, usdShaders here: https://github.com/PixarAnimationStudios/OpenUSD/tree/dev/pxr/usdImaging/plugin/usdShaders

Maybe the testPlug failure also points to that :/

[asluk]

Thanks @tallytalwar ! Is there something additional in PATH or PYTHONPATH etc that @anthony-linaro needs to do to detect the shaders?

@anthony-linaro , can you post the relevant bits from your PATH and PYTHONPATH when running tests?

[tallytalwar]

Ahh I see.. I think the "--no-imaging" arg to the build script doesn't build pxr/usdImaging which is where these shader plugins live.

From the PR description:

python build_scripts/build_usd.py --no-python --onetbb --tests --no-imaging C:/WoA/OpenUSD/build_dir

I also had to disable imaging, as building boost appears to be broken when using newer CMake versions (similar issue to https://github.com/PixarAnimationStudios/OpenUSD/issues/3285)

[asluk]

Thanks @tallytalwar .... @anthony-linaro , those failures are expected then ... @tallytalwar , is it worth a separate PR disabling some cases in the usdchecker tests when Imaging is not enabled in the build?

[tallytalwar]

@asluk yes definitely. I was planning to do that, but a PR will also be appreciated. Will sync it in for internal consumption promptly.

[anthony-linaro]

So, we would consider this ready to go then? Or is there a requirement to fix the imaging first?

[asluk]

So, we would consider this ready to go then? Or is there a requirement to fix the imaging first?

@anthony-linaro can you try running just testPlug to see if you can get more output from it? something like ctest -R testPlug

Overall I think this is generally ready to go, but defer to @sunyab as to next steps. Thanks all!

[anthony-linaro]

@asluk Full output:

C:\WoA\OpenUSD\build_dir\build\OpenUSD>ctest -C Release -R testPlug --output-on-failure
Test project C:/WoA/OpenUSD/build_dir/build/OpenUSD
    Start 226: testPlug
1/1 Test #226: testPlug .........................***Failed    0.21 sec
Warning: in _DeclareAliases at line 526 of C:\WoA\OpenUSD\pxr\base\plug\plugin.cpp -- Expected string for alias name, but found int
.Error: return code 3221225477 doesn't match expected 0 (EXPECTED_RETURN_CODE).chdir: C:\Users\Anthony\AppData\Local\Temp\tmp4oyg3b4d
cmd: ['C:/Program', 'Files/Python311-arm64/python.exe', 'C:/WoA/OpenUSD/build_dir/tests/testPlug']


0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.31 sec

The following tests FAILED:
        226 - testPlug (Failed)
Errors while running CTest

[asluk]

@asluk Full output:

C:\WoA\OpenUSD\build_dir\build\OpenUSD>ctest -C Release -R testPlug --output-on-failure
Test project C:/WoA/OpenUSD/build_dir/build/OpenUSD
    Start 226: testPlug
1/1 Test #226: testPlug .........................***Failed    0.21 sec
Warning: in _DeclareAliases at line 526 of C:\WoA\OpenUSD\pxr\base\plug\plugin.cpp -- Expected string for alias name, but found int
.Error: return code 3221225477 doesn't match expected 0 (EXPECTED_RETURN_CODE).chdir: C:\Users\Anthony\AppData\Local\Temp\tmp4oyg3b4d
cmd: ['C:/Program', 'Files/Python311-arm64/python.exe', 'C:/WoA/OpenUSD/build_dir/tests/testPlug']


0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.31 sec

The following tests FAILED:
        226 - testPlug (Failed)
Errors while running CTest

Thanks @anthony-linaro -- I can't think of something that would cause this, and I don't think that the warning is what is causing the failure. Also I'd expect many more tests to be broken if plugins were broken.

@sunyab , any ideas on how we can get more diagnostics on the failure? Maybe run with TF_DEBUG=PLUG* in the environment? Thanks!

[asluk]

@anthony-linaro can you set TF_DEBUG to be "PLUG*" in the environment ? Here is my LastTest.log file that gets output from ctest, when running the test successfully on Windows x86_64

LastTest.log