llvm
Allow specifying libcxx builder image.
This change attempts to shift the libc++ builders over to new backend infrastructure that allows running an arbitrary container for the libc++ job.
This has been a long time in the making, and support from github and gke is finally at the point where it's possible (hopefully).
This change should also demonstrate another important property: No Downtime Upgrades.
If this goes well, we'll be able to test the upgrade as a part of the PR process, and then commiting it to main should (ideally) not break anything.
@llvm/pr-subscribers-lldb @llvm/pr-subscribers-libcxx
@llvm/pr-subscribers-github-workflow
Author: Eric (EricWF)
Changes
This change attempts to shift the libc++ builders over to new backend infrastructure that allows running an arbitrary container for the libc++ job.
This has been a long time in the making, and support from github and gke is finally at the point where it's possible (hopefully).
This change should also demonstrate another important property: No Downtime Upgrades.
If this goes well, we'll be able to test the upgrade as a part of the PR process, and then commiting it to main should (ideally) not break anything.
Full diff: https://github.com/llvm/llvm-project/pull/110303.diff
1 Files Affected:
- (modified) .github/workflows/libcxx-build-and-test.yaml (+11-8)
diff --git a/.github/workflows/libcxx-build-and-test.yaml b/.github/workflows/libcxx-build-and-test.yaml
index b5e60781e00064..64855dad7197da 100644
--- a/.github/workflows/libcxx-build-and-test.yaml
+++ b/.github/workflows/libcxx-build-and-test.yaml
@@ -49,7 +49,8 @@ env:
jobs:
stage1:
if: github.repository_owner == 'llvm'
- runs-on: libcxx-runners-8-set
+ runs-on: libcxx-runners-set
+ container: ghcr.io/libcxx/actions-builder:testing-2024-09-21
continue-on-error: false
strategy:
fail-fast: false
@@ -84,7 +85,8 @@ jobs:
**/crash_diagnostics/*
stage2:
if: github.repository_owner == 'llvm'
- runs-on: libcxx-runners-8-set
+ runs-on: libcxx-runners-set
+ container: ghcr.io/libcxx/actions-builder:testing-2024-09-21
needs: [ stage1 ]
continue-on-error: false
strategy:
@@ -160,20 +162,21 @@ jobs:
'benchmarks',
'bootstrapping-build'
]
- machine: [ 'libcxx-runners-8-set' ]
+ machine: [ 'libcxx-runners-set' ]
include:
- config: 'generic-cxx26'
- machine: libcxx-runners-8-set
+ machine: libcxx-runners-set
- config: 'generic-asan'
- machine: libcxx-runners-8-set
+ machine: libcxx-runners-set
- config: 'generic-tsan'
- machine: libcxx-runners-8-set
+ machine: libcxx-runners-set
- config: 'generic-ubsan'
- machine: libcxx-runners-8-set
+ machine: libcxx-runners-set
# Use a larger machine for MSAN to avoid timeout and memory allocation issues.
- config: 'generic-msan'
- machine: libcxx-runners-8-set
+ machine: libcxx-runners-set
runs-on: ${{ matrix.machine }}
+ container: ghcr.io/libcxx/actions-builder:testing-2024-09-21
steps:
- uses: actions/checkout@v4
- name: ${{ matrix.config }}
This is amazing!
About the tests, I am not certain why the transitive includes test started failing, but I ran into something similar in https://github.com/llvm/llvm-project/pull/109720. I think this may be how we're running awk or something like that (I wasn't able to reproduce).
@EricWF I am trying to debug and fix this CI failure in https://github.com/llvm/llvm-project/pull/110554
:white_check_mark: With the latest revision this PR passed the C/C++ code formatter.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
How do I constantly fudge up my git history....
Fixing and force-pushing shortly.
but keep one cluster around for a few weeks just to finish running existing patches
I'll see what I can do. I don't know that I can muster enough capacity to make that happen. We may need to ask them to rebase their changes.
@Michael137 The LLDB data formatter tests are failing with this patch, see https://github.com/llvm/llvm-project/actions/runs/11484654125/job/31974962709?pr=110303. Gist of the error:
AssertionError: False is not true : launch failed (Cannot launch '/__w/.../lldb-dap/stackTrace/subtleFrames/TestDAP_subtleFrames.test_subtleFrames/a.out': personality set failed: Operation not permitted)
Full error
-- Testing: 1208 tests, 32 workers --
Testing: 0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
FAIL: lldb-api :: tools/lldb-dap/stackTrace/subtleFrames/TestDAP_subtleFrames.py (1207 of 1208)
******************** TEST 'lldb-api :: tools/lldb-dap/stackTrace/subtleFrames/TestDAP_subtleFrames.py' FAILED ********************
Script:
--
/usr/bin/python3.10 /__w/llvm-project/llvm-project/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env LLVM_LIBS_DIR=/__w/llvm-project/llvm-project/build/bootstrapping-build/./lib --env LLVM_INCLUDE_DIR=/__w/llvm-project/llvm-project/build/bootstrapping-build/include --env LLVM_TOOLS_DIR=/__w/llvm-project/llvm-project/build/bootstrapping-build/./bin --libcxx-include-dir /__w/llvm-project/llvm-project/build/bootstrapping-build/include/c++/v1 --libcxx-include-target-dir /__w/llvm-project/llvm-project/build/bootstrapping-build/include/x86_64-pc-linux-gnu/c++/v1 --libcxx-library-dir /__w/llvm-project/llvm-project/build/bootstrapping-build/./lib/x86_64-pc-linux-gnu --arch x86_64 --build-dir /__w/llvm-project/llvm-project/build/bootstrapping-build/lldb-test-build.noindex --lldb-module-cache-dir /__w/llvm-project/llvm-project/build/bootstrapping-build/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /__w/llvm-project/llvm-project/build/bootstrapping-build/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /__w/llvm-project/llvm-project/build/bootstrapping-build/./bin/lldb --compiler /__w/llvm-project/llvm-project/build/bootstrapping-build/./bin/clang --dsymutil /__w/llvm-project/llvm-project/build/bootstrapping-build/./bin/dsymutil --make /usr/bin/make --llvm-tools-dir /__w/llvm-project/llvm-project/build/bootstrapping-build/./bin --lldb-obj-root /__w/llvm-project/llvm-project/build/bootstrapping-build/tools/lldb --lldb-libs-dir /__w/llvm-project/llvm-project/build/bootstrapping-build/./lib --category libc++ /__w/llvm-project/llvm-project/lldb/test/API/tools/lldb-dap/stackTrace/subtleFrames -p TestDAP_subtleFrames.py
--
Exit Code: 1
Command Output (stdout):
--
lldb version 20.0.0git (https://github.com/llvm/llvm-project revision f114c52ffa919748cbe034d6e9acd971aa2d783a)
clang revision f114c52ffa919748cbe034d6e9acd971aa2d783a
llvm revision f114c52ffa919748cbe034d6e9acd971aa2d783a
Skipping the following test categories: ['dsym', 'gmodules', 'debugserver', 'objc']
========= DEBUG ADAPTER PROTOCOL LOGS =========
1729721465.380973816 -->
Content-Length: 344
{
"arguments": {
"adapterID": "lldb-native",
"clientID": "vscode",
"columnsStartAt1": true,
"linesStartAt1": true,
"locale": "en-us",
"pathFormat": "path",
"sourceInitFile": false,
"supportsRunInTerminalRequest": true,
"supportsStartDebuggingRequest": true,
"supportsVariablePaging": true,
"supportsVariableType": true
},
"command": "initialize",
"seq": 1,
"type": "request"
}
1729721465.383299589 <--
Content-Length: 1585
{
"body": {
"__lldb": {
"version": "lldb version 20.0.0git (https://github.com/llvm/llvm-project revision f114c52ffa919748cbe034d6e9acd971aa2d783a)\n clang revision f114c52ffa919748cbe034d6e9acd971aa2d783a\n llvm revision f114c52ffa919748cbe034d6e9acd971aa2d783a"
},
"completionTriggerCharacters": [
".",
" ",
"\t"
],
"exceptionBreakpointFilters": [
{
"default": false,
"filter": "cpp_catch",
"label": "C++ Catch"
},
{
"default": false,
"filter": "cpp_throw",
"label": "C++ Throw"
},
{
"default": false,
"filter": "objc_catch",
"label": "Objective-C Catch"
},
{
"default": false,
"filter": "objc_throw",
"label": "Objective-C Throw"
}
],
"supportTerminateDebuggee": true,
"supportsCompletionsRequest": true,
"supportsConditionalBreakpoints": true,
"supportsConfigurationDoneRequest": true,
"supportsDataBreakpoints": true,
"supportsDelayedStackTraceLoading": true,
"supportsDisassembleRequest": true,
"supportsEvaluateForHovers": true,
"supportsExceptionInfoRequest": true,
"supportsExceptionOptions": true,
"supportsFunctionBreakpoints": true,
"supportsGotoTargetsRequest": false,
"supportsHitConditionalBreakpoints": true,
"supportsInstructionBreakpoints": true,
"supportsLoadedSourcesRequest": false,
"supportsLogPoints": true,
"supportsModulesRequest": true,
"supportsProgressReporting": true,
"supportsReadMemoryRequest": true,
"supportsRestartFrame": false,
"supportsRestartRequest": true,
"supportsRunInTerminalRequest": true,
"supportsSetVariable": true,
"supportsStepBack": false,
"supportsStepInTargetsRequest": true,
"supportsSteppingGranularity": true,
"supportsValueFormattingOptions": true
},
"command": "initialize",
"request_seq": 1,
"seq": 0,
"success": true,
"type": "response"
}
1729721465.383833170 -->
Content-Length: 900
{
"arguments": {
"commandEscapePrefix": null,
"disableASLR": true,
"displayExtendedBacktrace": false,
"enableAutoVariableSummaries": false,
"enableSyntheticChildDebugging": false,
"initCommands": [
"settings clear -all",
"settings set symbols.enable-external-lookup false",
"settings set target.inherit-tcc true",
"settings set target.disable-aslr false",
"settings set target.detach-on-error false",
"settings set target.auto-apply-fixits false",
"settings set plugin.process.gdb-remote.packet-timeout 60",
"settings set symbols.clang-modules-cache-path \"/__w/llvm-project/llvm-project/build/bootstrapping-build/lldb-test-build.noindex/module-cache-lldb/lldb-api\"",
"settings set use-color false"
],
"program": "/__w/llvm-project/llvm-project/build/bootstrapping-build/lldb-test-build.noindex/tools/lldb-dap/stackTrace/subtleFrames/TestDAP_subtleFrames.test_subtleFrames/a.out"
},
"command": "launch",
"seq": 2,
"type": "request"
}
1729721465.384315491 <--
Content-Length: 106
{
"body": {
"category": "console",
"output": "Running initCommands:\n"
},
"event": "output",
"seq": 0,
"type": "event"
}
1729721465.384341478 <--
Content-Length: 111
{
"body": {
"category": "console",
"output": "(lldb) settings clear -all\n"
},
"event": "output",
"seq": 0,
"type": "event"
}
1729721465.384360790 <--
Content-Length: 141
{
"body": {
"category": "console",
"output": "(lldb) settings set symbols.enable-external-lookup false\n"
},
"event": "output",
"seq": 0,
"type": "event"
}
1729721465.384387016 <--
Content-Length: 128
{
"body": {
"category": "console",
"output": "(lldb) settings set target.inherit-tcc true\n"
},
"event": "output",
"seq": 0,
"type": "event"
}
1729721465.384403944 <--
Content-Length: 130
{
"body": {
"category": "console",
"output": "(lldb) settings set target.disable-aslr false\n"
},
"event": "output",
"seq": 0,
"type": "event"
}
1729721465.384418964 <--
Content-Length: 133
{
"body": {
"category": "console",
"output": "(lldb) settings set target.detach-on-error false\n"
},
"event": "output",
"seq": 0,
"type": "event"
}
1729721465.384435415 <--
Content-Length: 135
{
"body": {
"category": "console",
"output": "(lldb) settings set target.auto-apply-fixits false\n"
},
"event": "output",
"seq": 0,
"type": "event"
}
1729721465.384450674 <--
Content-Length: 148
{
"body": {
"category": "console",
"output": "(lldb) settings set plugin.process.gdb-remote.packet-timeout 60\n"
},
"event": "output",
"seq": 0,
"type": "event"
}
1729721465.384467602 <--
Content-Length: 249
{
"body": {
"category": "console",
"output": "(lldb) settings set symbols.clang-modules-cache-path \"/__w/llvm-project/llvm-project/build/bootstrapping-build/lldb-test-build.noindex/module-cache-lldb/lldb-api\"\n"
},
"event": "output",
"seq": 0,
"type": "event"
}
1729721465.384484768 <--
Content-Length: 120
{
"body": {
"category": "console",
"output": "(lldb) settings set use-color false\n"
},
"event": "output",
"seq": 0,
"type": "event"
}
1729721465.475079060 <--
Content-Length: 320
{
"command": "launch",
"message": "Cannot launch '/__w/llvm-project/llvm-project/build/bootstrapping-build/lldb-test-build.noindex/tools/lldb-dap/stackTrace/subtleFrames/TestDAP_subtleFrames.test_subtleFrames/a.out': personality set failed: Operation not permitted",
"request_seq": 2,
"seq": 0,
"success": false,
"type": "response"
}
1729721465.475131273 <--
Content-Length: 46
{
"event": "initialized",
"seq": 0,
"type": "event"
}
1729721465.482665062 -->
Content-Length: 88
{
"arguments": {
"terminateDebuggee": true
},
"command": "disconnect",
"seq": 3,
"type": "request"
}
1729721465.483467817 <--
Content-Length: 1290
{
"event": "terminated",
"seq": 0,
"statistics": {
"commands": "{\"settings clear\":1}",
"memory": "{\"strings\":{\"bytesTotal\":1011712,\"bytesUnused\":977290,\"bytesUsed\":34422}}",
"targets": "[{\"breakpoints\":[],\"dyldPluginName\":\"posix-dyld\",\"expressionEvaluation\":{\"failures\":0,\"successes\":0},\"frameVariable\":{\"failures\":0,\"successes\":0},\"launchOrAttachTime\":0.093175982000000004,\"moduleIdentifiers\":[93901650622048],\"signals\":[],\"sourceMapDeduceCount\":0,\"sourceRealpathAttemptCount\":0,\"sourceRealpathCompatibleCount\":0,\"stopCount\":1,\"summaryProviderStatistics\":[],\"targetCreateTime\":0.00026899999999999998,\"totalBreakpointResolveTime\":0,\"totalSharedLibraryEventHitCount\":0}]",
"totalDebugInfoByteSize": 105315,
"totalDebugInfoEnabled": 1,
"totalDebugInfoIndexLoadedFromCache": 0,
"totalDebugInfoIndexSavedToCache": 0,
"totalDebugInfoIndexTime": 0.0022790000000000002,
"totalDebugInfoParseTime": 0.00053600000000000002,
"totalModuleCount": 1,
"totalModuleCountHasDebugInfo": 1,
"totalModuleCountWithIncompleteTypes": 0,
"totalModuleCountWithVariableErrors": 0,
"totalSymbolTableIndexTime": 0.00069200000000000002,
"totalSymbolTableParseTime": 0.00090600000000000001,
"totalSymbolTableStripped": 0,
"totalSymbolTablesLoadedFromCache": 0,
"totalSymbolTablesSavedToCache": 0
},
"type": "event"
}
1729721465.483519793 <--
Content-Length: 81
{
"command": "disconnect",
"request_seq": 3,
"seq": 0,
"success": true,
"type": "response"
}
========= END =========
--
Command Output (stderr):
--
FAIL: LLDB (/__w/llvm-project/llvm-project/build/bootstrapping-build/bin/clang-x86_64) :: test_subtleFrames (TestDAP_subtleFrames.TestDAP_subtleFrames)
======================================================================
FAIL: test_subtleFrames (TestDAP_subtleFrames.TestDAP_subtleFrames)
Internal stack frames (such as the ones used by `std::function`) are marked as "subtle".
----------------------------------------------------------------------
Traceback (most recent call last):
File "/__w/llvm-project/llvm-project/lldb/test/API/tools/lldb-dap/stackTrace/subtleFrames/TestDAP_subtleFrames.py", line 20, in test_subtleFrames
self.build_and_launch(program)
File "/__w/llvm-project/llvm-project/lldb/packages/Python/lldbsuite/test/tools/lldb-dap/lldbdap_testcase.py", line 485, in build_and_launch
return self.launch(
File "/__w/llvm-project/llvm-project/lldb/packages/Python/lldbsuite/test/tools/lldb-dap/lldbdap_testcase.py", line 442, in launch
self.assertTrue(
AssertionError: False is not true : launch failed (Cannot launch '/__w/llvm-project/llvm-project/build/bootstrapping-build/lldb-test-build.noindex/tools/lldb-dap/stackTrace/subtleFrames/TestDAP_subtleFrames.test_subtleFrames/a.out': personality set failed: Operation not permitted)
Config=x86_64-/__w/llvm-project/llvm-project/build/bootstrapping-build/bin/clang
----------------------------------------------------------------------
Ran 1 test in 1.726s
FAILED (failures=1)
Failures per category:
libc++ - 1
lldb-dap - 1
--
********************
Testing: 0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
********************
Failed Tests (1):
lldb-api :: tools/lldb-dap/stackTrace/subtleFrames/TestDAP_subtleFrames.py
What's special about this patch is that we're running the CI inside a Docker VM that's running inside another Docker VM. Do you have any ideas as to why this might have started failing?
CC @vogelsgesang @walter-erquinigo since they're more familiar with the DAP infrastructure than me. Any idea why we might fail to launch the test executable in a container (inside a container)?
Interestingly, looks like @mordante has run into this before with the data-formatter CI:
- https://discourse.llvm.org/t/running-lldb-in-a-container/76801
- https://github.com/llvm/llvm-project/pull/88312
Though the fix for it doesn't seem to work for this case?
Any idea why we might fail to launch the test executable in a container (inside a container)?
You need to set up the debug session with settings set target.disable-aslr false so that you don't have that problem with the personality syscall. This can be done here https://github.com/llvm/llvm-project/blob/main/lldb/packages/Python/lldbsuite/test/tools/lldb-dap/dap_server.py#L758 so that all launch requests use the correct setting. See that there's a disableASLR setting that you can try to have a default value for.
Any idea why we might fail to launch the test executable in a container (inside a container)?
You need to set up the debug session with
settings set target.disable-aslr falseso that you don't have that problem with the personality syscall. This can be done here https://github.com/llvm/llvm-project/blob/main/lldb/packages/Python/lldbsuite/test/tools/lldb-dap/dap_server.py#L758 so that all launch requests use the correct setting. See that there's a disableASLR setting that you can try to have a default value for.
Given we already flipped the default to settings set target.disable-aslr false for all non-DAP tests, could we set disableASLR to False here too?
The behaviour is a bit inconsistent at the moment. From the logs:
{
"arguments": {
"commandEscapePrefix": null,
"disableASLR": true,
....
"initCommands": [
...
"settings set target.disable-aslr false",
Thanks a lot for chiming in! Side question: I thought ASLR was the problem, so shouldn't the solution be to turn off ASLR, i.e. use settings set target.disable-aslr true? The Discord thread seems to have slightly conflicting information.
Thanks a lot for chiming in! Side question: I thought ASLR was the problem, so shouldn't the solution be to turn off ASLR, i.e. use
settings set target.disable-aslr true? The Discord thread seems to have slightly conflicting information.
Ah good point. I think what's happening is that we're failing to disable ASLR, hence the recommendation in that thread is to settings set target.disable-aslr false. But we don't do that for DAP tests. As Jason pointed out in that thread, having target.disable-aslr true isn't very useful in the testsuite, so we should be fine to adopt the same for the DAP tests (i.e., have disableASLR=False).
https://github.com/llvm/llvm-project/pull/113593
If you want to make sure that this definitely resolves the issue @EricWF, you could cherry-pick that change and let the PR CI run again?
Thanks for your fast help with this. Re-basing and rerunning now.
Np! Haven't merged it yet though. Just waiting for CI to pass
Hmm am I reading this right that the latest run still failed, despite the cherry-pick? EDIT: Oh nvm, the change didn't seem to kick in yet:
{
"arguments": {
"commandEscapePrefix": null,
"disableASLR": true,
"displayExtendedBacktrace": false,
"enableAutoVariableSummaries": false,
"enableSyntheticChildDebugging": false,
"initCommands": [
"settings clear -all",
"settings set symbols.enable-external-lookup false",
"settings set target.inherit-tcc true",
"settings set target.disable-aslr false",
Ignore me
FYI, had to adjust the flag in one other place. Feel free to rebase the branch on main. I merged the changes. Let me know if the CI still fails
It looks like it's still failing with the latest run :-(
Argh that's unfortunate. How about we skip this test in libc++ CI to unblock this PR and I'll open a github issue to re-enable the test?
@EricWF It's probably easiest if you just add --filter-out=TestDAP_subtleFrames.py to the following LIT invocation as part of this PR: https://github.com/llvm/llvm-project/blob/8c4bc1e75de27adfbaead34b895b0efbaf17bd02/libcxx/utils/ci/run-buildbot#L397
But if you prefer me doing it separately, let me know.
It looks like it's still failing with the latest run :-(
Argh that's unfortunate. How about we skip this test in libc++ CI to unblock this PR and I'll open a github issue to re-enable the test?
@EricWF It's probably easiest if you just add
--filter-out=TestDAP_subtleFrames.pyto the following LIT invocation as part of this PR:https://github.com/llvm/llvm-project/blob/8c4bc1e75de27adfbaead34b895b0efbaf17bd02/libcxx/utils/ci/run-buildbot#L397
But if you prefer me doing it separately, let me know.
I have concerns about using the run-buildbot file to hide failing tests. I'll hold off on this change a little longer.
It looks like it's still failing with the latest run :-(
Argh that's unfortunate. How about we skip this test in libc++ CI to unblock this PR and I'll open a github issue to re-enable the test? @EricWF It's probably easiest if you just add
--filter-out=TestDAP_subtleFrames.pyto the following LIT invocation as part of this PR: https://github.com/llvm/llvm-project/blob/8c4bc1e75de27adfbaead34b895b0efbaf17bd02/libcxx/utils/ci/run-buildbot#L397But if you prefer me doing it separately, let me know.
I have concerns about using the
run-buildbotfile to hide failing tests. I'll hold off on this change a little longer.
That's fair. In that case, @walter-erquinigo @clayborg Do you have any ideas on how to best debug this?
Summary: the TestDAP_subtleFrames.py test is failing when run in a container (in a container):
AssertionError: False is not true : launch failed (Cannot launch '/__w/.../lldb-dap/stackTrace/subtleFrames/TestDAP_subtleFrames.test_subtleFrames/a.out': personality set failed: Operation not permitted)
Our theory was that this happened when trying to disable ASLR. So we're no longer doing that for the DAP tests. But we're still failing with the above.
I'll try raise a draft PR that mimics this but with some additional LLDB logging.
Hmm so I opened a draft PR with this change and explicitly set disableASLR on the DAP server and the tests seemed to pass: https://github.com/llvm/llvm-project/pull/113891
With server patch: https://github.com/llvm/llvm-project/actions/runs/11552969549/job/32154860810?pr=113891 Without server patch: https://github.com/llvm/llvm-project/actions/runs/11557069616/job/32168537938?pr=113891
So it does look like this is still disableASLR related. I don't know why simply passing disableASLR to the server isn't doing the expected thing. Will have to investigate...
Ooh that's because it's hardcoded in the lldb-dap executable: https://github.com/llvm/llvm-project/blob/f14743794587db102c6d1b20f9c87a1ac20decfd/lldb/tools/lldb-dap/lldb-dap.cpp#L2103-L2104
Fix should be simple enough. Just need to always pass the disableASLR value from Python, regardless of whether it's set to True or False
Just merged the fix. Let me know if you're still facing issues after the rebase
@EricWF The CI failures are unrelated issues on main that have been fixed. It looks like everything is working now.
I'll let you merge this since you likely want to adjust the capacity and other stuff before or closely after you merge, but as far as I'm concerned this is good to go. Thanks a whole lot for this improvement!