Subprojects / Multi-project builds

[eatdrinksleepcode]

One of the things that Gradle and Maven both do well that few others do is multi-project builds. Tasks like "compile", "test", and "pack" are defined per project rather than once at the root (although you can define them once in a way that applies to all subprojects). In large solutions, the ability to run tasks for particular projects rather than the entire solution is critical to efficient developer productivity. As an example in Gradle, when I am developing in a particular subproject, I can run the tests for just that project by executing gradle myproject:test.

I think this is actually one of the most challenging problems for a .NET build system to solve, because of our reliance on MSBuild. In Gradle, dependencies between projects are expressed in the build script, allowing Gradle to ensure that all dependent projects are built. For example, if project B depends on project A, and I execute gradle B:compile, Gradle knows that A:compile must run first and that the artifacts from A must be on the compilation classpath for B. This ensures that any dependencies of A:compile are also run, and that B has what it needs to compile, all from a single dependency declaration. But in .NET, dependencies between projects are expressed as project references in the csproj file. We can't realistically break that convention, as the majority of developers rely on the csproj for their IDE integration (Visual Studio / SharpDevelop / Xamarin Studio). The new project system (project.json) may improve that situation somewhat, but we should assume that the MSBuild format will remain dominant for some period of time. Which means that dependencies are declared in a separate file. This creates two problems:

1. Creating a Gradle-like dependency graph in Bau will require either forcing dependencies to be duplicated in both the csproj and the Bau project file, or Bau will have to read the MSBuild project file and interpret the references (or some other solution to this that I haven't thought of; CoreFX does some interesting things with Nuget dependencies and references from within MSBuild that are worth investigating).
2. Using MSBuild to build a project will automatically build referenced projects as well outside of Bau, meaning that there is no way from within Bau to ensure that dependencies of referenced build tasks will be executed. (There is a MSBuild property called BuildProjectReferences which may help with this).

Even with these potential issues in the way, I think the advantage of clean multi-project build support are too great to ignore.

[adamralph]

Interesting.

A naive question: if I have A.csproj which declares B.csproj as a dependency, does MSBuild not know to build B.csproj if I tell it to build A.csproj?

[eatdrinksleepcode]

It certainly does. However, when MSBuild builds B.csproj, Bau doesn't know that it's doing that. So we can't build any kind of workflow that involves building B in Bau (such as generating files before the build).

The trick of course is that even if we do build such a workflow, neither Visual Studio nor any other .NET IDE will be aware of it.

I am working on building a basic plugin on top of the currently existing API to demonstrate the concept and determine more specifically what the gaps are.

[aarondandy]

So if I understand you correctly, you want to explore an integration between msbuild targets and bau tasks, like interweaving them? I know a few people to ask, I'll see if they have any thoughts on that. Also another thing to consider is that msbuild may end up as a legacy tool in a few years so there may be some real benefit to this kind of thinking.

[eatdrinksleepcode]

So if I understand you correctly, you want to explore an integration between msbuild targets and bau tasks, like interweaving them?

Not exactly. Let me try to illustrate with an example. Imagine the Bau build file looked, in part, something like this:

bau.Subproject("core", "src/Bau")
    .MSBuild("build").Do(msb =>
    {
        msb.Solution = "Bau.csproj";
        ...
    })
    .Exec("pack").DependsOn("build").Do(exec =>
    {
        exec.Run(nugetCommand)
            .With(
                "pack", "Bau.nuspec",
                ...
            );
    });

I could then run build core:pack, and it would run build and pack for just the core project. Now I add a similar definition for Bau.Exec:

bau.Subproject("exec", "src/Bau.Exec")
    .MSBuild("build").Do(msb =>
    {
        msb.Solution = "Bau.Exec.csproj";
        ...
    })
    .Exec("pack").DependsOn("build").Do(exec =>
    {
        exec.Run(nugetCommand)
            .With(
                "pack", "Bau.nuspec",
                ...
            );
    });

Now I can run build exec:pack, and it will build and pack the exec project. This level of granularity doesn't matter much for a small solution like Bau, where we can quickly build the entire solution; but it is a big win for large multi-project solutions, where invoking tasks specifically related to the part of the code that I am currently working on is much more efficient.

However, the downside to this is that building the exec project with MSBuild will also cause the core project to be compiled, but in a way that Bau has no visibility to. This is unintuitive to me as the user, since I may have defined other workflow in my Bau build script related to "core:build" (for example I might make it depend on another task that generates source files before the build). I would expect that any time core is built, the dependencies that I have defined will run first. But in this case they wouldn't. I can work around this by telling MSBuild not to build dependent projects, and defining those dependencies in my Bau build script instead:

bau.Subproject("exec", "src/Bau.Exec")
    .MSBuild("build").DependsOn("core:build").Do(msb =>
    {
        msb.Solution = "Bau.Exec.csproj";
        msb.Properties = new { BuildProjectReferences = false };
        ...
    })

Now, when I run "build exec:build", Bau will run "core:build" - and any of its dependencies - first, and then run "exec:build", but without attempting to re-compile core. This is exactly what we want, except that we are now required to specify dependencies between projects at the task level, which is redundant with the project references that already exist in the csproj file. If we add a new reference to the csproj file in Visual Studio, but we don't update the Bau build script, then our compile will fail at best, or worse will use an outdated build of the new referenced project, probably without us realizing it, leading to bugs and strange behavior that will be difficult to track down. We may able to mitigate this by inspecting the csproj file and determining build dependencies automatically; but I haven't attempted to prove that out yet.

[aarondandy]

To reduce redundancy I wonder if a simpler approach would be to integrate Bau into each msbuild project using a tool. Something as simple as bau.Task("core.csproj:before") and bau install core.csproj may take us pretty far and allow for integration with tools that revolve around MSBuild, like visual studio and well... MSBuild itself.

[eatdrinksleepcode]

@aarondandy While that might make it easier to define "before" steps in MSBuild, I worry that the integration would be unintuitive when invoked from Bau. Using our previous example, if I invoke exec:build, and it invokes MSBuild to build the exec project, which builds the core project, which invokes the "before" task in Bau, will this second instance of Bau have any of the context of the original instance of Bau (such as what tasks have already been invoked)? If not I think it would be difficult for me as a user to reason about my build. I also think this awareness would be difficult to achieve, unless we move the execution of MSBuild in-process, which may or may not be feasible.

I still think our best bet is to not rely on MSBuild's automatic dependency building (by setting BuildProjectReferences=false), but instead read the dependency information from the csproj file and use to create the task dependency graph in Bau.

[eatdrinksleepcode]

As I have thought more about this and done some spiking using Bau's build, subprojects don't necessarily have a lot of value on their own. After all, granular tasks such as in my example above can be done using the current API and a naming convention; you don't need subprojects for that. Subprojects don't really start to show their value until you start adding in other features that make creating dependency graphs easier. I am going to log some of those potential features separately and reference this issue.

[eatdrinksleepcode]

Still thinking about this...one of the big benefits of subprojects is that they create a convenient scope for applying plugins, such as #220. The plugin can use the configuration of the project (most importantly its location) to configure itself automatically.

It might help if I make the discussion a little less abstract. If we combine this issue with #219, #220, #222, and a few other things, I can imagine the Bau build file someday looking like this:

// parameters
var versionSuffix = Environment.GetEnvironmentVariable("VERSION_SUFFIX");
var nugetVerbosity = Environment.GetEnvironmentVariable("NUGET_VERBOSITY") ?? "quiet";

// solution specific variables
var version = File.ReadAllText("src/CommonAssemblyInfo.cs").Split(new[] { "AssemblyInformationalVersion(\"" }, 2, StringSplitOptions.None).ElementAt(1).Split(new[] { '"' }).First();

// solution agnostic tasks
var bau = Require<Bau>();

bau
.ForAll<MSBuild>(msb => {
    msb.MSBuildVersion = "net45";
    msb.Verbosity = bau.GetProperty<Verbosity>("msBuildFileVerbosity");
})
.ForAll<NuGetTask>(n => {
    n.Verbosity = nugetVerbosity;
})
.ForAll<Pack>(n => {
    n.Version = version + versionSuffix;
    n.Property("Configuration", "Release");
})
.Project("bau").At("src/Bau").CSProj().NuGet()
.Project("exec").At("src/Bau.Exec").CSProj().NuGet()
.Project("xunit".At("src/Bau.Xunit").CSProj().NuGet()
.Project("test").At("src/test").With(p =>
    p
    .ForAll<Xunit>(x => {
        x.Html().Xml();
    })
    .Project("unit").At("Bau.Test.Unit").CSProj().Xunit()
    .Project("component").At("Bau.Test.Component").CSProj().Xunit()
    .Project("acceptance").At("Bau.Test.Acceptance").CSProj().Xunit()
)
.NuGetRestore("src/Bau.sln")
;

bau.Run();

There are ways it could get even more succinct, but I hope that gives you the idea of where I imagine Bau going.