Plugin build fails with `error: buildFailed` when benchmarking with a recent main-snapshot toolchain

[clackary]

We've working on mainlining some optimizations to Swift. I'm trying to run benchmarks on an internal project using a toolchain with the optimizations and without to measure the performance impact.

The plan was to establish a baseline by running benchmarks using a main-snapshot toolchain, and then building a custom toolchain with our optimizations cherrypicked onto that tag. With the main-snapshot, I ran into a pretty vague build failure when building the plugin.

$ swift package --disable-sandbox --package-path Benchmarks benchmark                      
Building for debugging...
[5/5] Applying BenchmarkTool-tool
Build of product 'BenchmarkTool' complete! (7.76s)
Build complete!
Building BenchmarkTool in release mode...
error: buildFailed

This also reproduces in the package-benchmarks Benchmarks sub-package.

Steps to reproduce:

1. Install a main-snapshot toolchain: swiftly install main-snapshot
2. Set that toolchain (may require overriding the 5.10 value in .swift-version: swiftly use main-snapshot
3. Attempt to benchmark in the Benchmarks sub-package: swift package --disable-sandbox --package-path Benchmarks benchmark

Initial triage: It looks like this check is the source of the error. This seems to be happening when the plugin build is successful, and no artifacts are registered builtArtifacts, as some additional logging shows that builtArtifacts is indeed empty.

It does not reproduce against a recent 6.2 snapshot.

[clackary]

My guess is that this either represents a regression or a change in behavior in swift-package-manager. I started the issue here because this code seems to be part of a workaround for another issue. It's still open, but if some other change also resolved it, the workaround could be dropped.

[hassila]

Thanks for the report - without digging further it seems perhaps it's some regression in main if it works up to 6.2 - out pfnprrixenfor a few weeks right now so can't dig in right now, but perhapsnworth checking if eg. Foundation has similar issues as they have a reasonably decent benchmark suite using the package