`WithCustomBuildConfiguration` breaks build

[timcassell]

Discussed in https://github.com/dotnet/BenchmarkDotNet/discussions/2424

^{Originally posted by SebastianStehle September 2, 2023} Hi,

I want to benchmark with local version of the library with previous nuget versions. Therefore I have created the following setup:

Code for that can be found at: https://github.com/SebastianStehle/mjml-net/blob/main/Mjml.Net.Benchmark/TemplateBenchmarks.cs

1. Define multiple jobs:

var baseJob = Job.ShortRun;

            AddJob(baseJob
                .WithId("Dev").WithBaseline(true));

            AddJob(baseJob.WithCustomBuildConfiguration("V1_24")
                .WithId("1.24.0"));

            AddJob(baseJob.WithCustomBuildConfiguration("V2_0")
                .WithId("2.0.0"));

            AddJob(baseJob.WithCustomBuildConfiguration("V2_1")
                .WithId("2.1.0"));

2. Load the nuget version based on configuration:

<ItemGroup Condition="'$(Configuration)'=='V1_24'">
	<PackageReference Include="Mjml.Net" Version="1.24.0" />
  </ItemGroup>
	
  <ItemGroup Condition="'$(Configuration)'=='V2_0'">
	<PackageReference Include="Mjml.Net" Version="2.0.0" />
  </ItemGroup>
	
  <ItemGroup Condition="'$(Configuration)'=='V2_1'">
	<PackageReference Include="Mjml.Net" Version="2.1.0-beta1" />
  </ItemGroup>

  <ItemGroup Condition="'$(Configuration)'=='Debug'">
    <ProjectReference Include="..\Mjml.Net\Mjml.Net.csproj" />
  </ItemGroup>

  <ItemGroup Condition="'$(Configuration)'=='Release'">
    <ProjectReference Include="..\Mjml.Net\Mjml.Net.csproj" />
  </ItemGroup>

But when I run it I get errors like this:

// Execute: dotnet 8013b23c-cfa0-4303-a97b-ce66f7c64415.dll --anonymousPipes 1412 4632 --benchmarkName "Mjml.Net.Benchmarking.TemplateBenchmarks.Render_Template_Minify(MjmlTemplateFilePath: \"./Templates/Amario.mjml\")" --job 1.24.0 --benchmarkId 1 in D:\mjml\mjml-net\Mjml.Net.Benchmark\bin\Release\net7.0\8013b23c-cfa0-4303-a97b-ce66f7c64415\bin\V1_24\net7.0
// BeforeAnythingElse

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.IO.FileNotFoundException: Could not load file or assembly 'Mjml.Net, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
File name: 'Mjml.Net, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null'
   at Mjml.Net.Benchmarking.TemplateBenchmarks..ctor()
   at BenchmarkDotNet.Autogenerated.Runnable_1..ctor() in D:\mjml\mjml-net\Mjml.Net.Benchmark\bin\Release\net7.0\8013b23c-cfa0-4303-a97b-ce66f7c64415\8013b23c-cfa0-4303-a97b-ce66f7c64415.notcs:line 395
   at BenchmarkDotNet.Autogenerated.Runnable_1.Run(IHost host, String benchmarkName) in D:\mjml\mjml-net\Mjml.Net.Benchmark\bin\Release\net7.0\8013b23c-cfa0-4303-a97b-ce66f7c64415\8013b23c-cfa0-4303-a97b-ce66f7c64415.notcs:line 339
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in D:\mjml\mjml-net\Mjml.Net.Benchmark\bin\Release\net7.0\8013b23c-cfa0-4303-a97b-ce66f7c64415\8013b23c-cfa0-4303-a97b-ce66f7c64415.notcs:line 57
// AfterAll

For me it seems that it tries to build with the wrong version. If I check the bin folder, I actually see the 2.0.0 version of the dll. there, but if I build with dotnet build -c V1_24 the correct version is used.

It seems that i am doing something wrong.

[timcassell]

@viktorhofer Any idea why building these in parallel fails or is incorrect? Even with the changes in #2393 where binary/intermediate outputs are forced to different directories, it still fails.

[rainersigwald]

NuGet does not support PackageReferences that are conditional on anything other than TargetFramework.

This problem is the reason why: the project.assets.json file is written to the obj directory (not a config- or target-framework-specific subfolder thereof), so different configurations share the same assets file. That means that other conditions can work as long as they're restored and then built with no intermediate step. If you restore for one of the other configurations before building, you'll get the behavior you're describing.

There are two workarounds:

1. Build and restore sequentially so the assets file always has the right contents for the current configuration.
2. Move the assets file location to a place that is unique for your given configuration. The easiest way to do that is to set BaseIntermediateOutputPath in a Directory.Build.props.

Note that neither gets you to a supported configuration, but they can be made to work.

[timcassell]

@rainersigwald I tried setting BaseIntermediateOutputPath via the dotnet command, but when I do, I get errors.

// Build Error: Standard output:

 Standard error:
 MSBuild version 17.9.4+90725d08d for .NET
  Determining projects to restore...
  Restored C:\Users\Tim\Downloads\mjml-net-main\Mjml.Net.Benchmark\Mjml.Net.Benchmark.csproj (in 576 ms).
C:\Program Files\dotnet\sdk\8.0.201\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(266,5): error NETSDK1005: Assets file 'C:\Users\Tim\Downloads\mjml-net-main\Mjml.Net.Benchmark\bin\Release\net7.0\5f8e1062-55e6-4907-a09e-54dd45bb6b9e\obj\V2_1\net7.0\project.assets.json' doesn't have a target for 'netstandard2.0'. Ensure that restore has run and that you have included 'netstandard2.0' in the TargetFrameworks for your project. [C:\BenchmarkDotNet\src\BenchmarkDotNet.Annotations\BenchmarkDotNet.Annotations.csproj::TargetFramework=netstandard2.0]
Build FAILED.
C:\Program Files\dotnet\sdk\8.0.201\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(266,5): error NETSDK1005: Assets file 'C:\Users\Tim\Downloads\mjml-net-main\Mjml.Net.Benchmark\bin\Release\net7.0\5f8e1062-55e6-4907-a09e-54dd45bb6b9e\obj\V2_1\net7.0\project.assets.json' doesn't have a target for 'netstandard2.0'. Ensure that restore has run and that you have included 'netstandard2.0' in the TargetFrameworks for your project. [C:\BenchmarkDotNet\src\BenchmarkDotNet.Annotations\BenchmarkDotNet.Annotations.csproj::TargetFramework=netstandard2.0]
    0 Warning(s)
    1 Error(s)
Time Elapsed 00:00:04.00

.AppendArgument($"/p:IntermediateOutputPath=\"{artifactsPaths.IntermediateDirectoryPath}{Path.AltDirectorySeparatorChar}\"")
.AppendArgument($"/p:BaseIntermediateOutputPath=\"{artifactsPaths.IntermediateDirectoryPath}{Path.AltDirectorySeparatorChar}\"")
.AppendArgument($"/p:OutDir=\"{artifactsPaths.BinariesDirectoryPath}{Path.AltDirectorySeparatorChar}\"")
.AppendArgument($"/p:OutputPath=\"{artifactsPaths.BinariesDirectoryPath}{Path.AltDirectorySeparatorChar}\"")

Removing BaseIntermediateOutputPath and leaving IntermediateOutputPath, OutDir, and OutputPath it works fine when run sequentially. What's wrong with this method?

[timcassell]

Nevermind, I found my answer at https://github.com/dotnet/sdk/issues/2003#issuecomment-369408964. Setting ArtifactsPath instead works (though it requires sdk 8+).

[timcassell]

Well, setting ArtifactsPath seems to work for every runtime except wasm (and probably MonoAOTLLVM, but we don't have any tests for it). Might you have any insight on that @LoopedBard3? https://github.com/dotnet/BenchmarkDotNet/actions/runs/8696789893

[matouskozak]

Well, setting ArtifactsPath seems to work for every runtime except wasm (and probably MonoAOTLLVM, but we don't have any tests for it). Might you have any insight on that @LoopedBard3? https://github.com/dotnet/BenchmarkDotNet/actions/runs/8696789893

@timcassell do you have any more information about why the benchmarks are crashing. I took a look at the run you posted and there seems to be only mentions about "Toolchain Build Failure" without any specific information what failed during the build stage.

[timcassell]

@matouskozak Unfortunately I don't have any more details than what are in those logs. I can't run wasm benchmarks on my machine since it doesn't support windows.