Proposal: Add support for build target dependencies

[terids]

Description

The build system is very flexible is its current state when building for a single target at a time, with support for adding c3 library file dependencies in the project.json file with "dependency-search-paths": ["lib"]" and "dependencies": [ ].

A useful enhancement would be to allow targets to depend on other targets in the same project. This would prevent having to run multiple build commands when a project consists of multiple targets.

Use case

Consider the following project layout, where game is an executable:

Project
├── lib
│   └── lib_for_engine.c3l
│   └── lib_for_platform.c3l
├── game
│   └── src
│       └── main.c3
└── src
    ├── engine
    │   └── engine.c3
    └── platform
        └── platform.c3

To build the game and engine/platform code together, there are currently 2 obvious options:

1. Compile everything as one target

This requires adding src/** to the game sources, and adding all required engine/platform dependencies to each relevant game target configuration (release, debug, etc.)

In the example above, the game target would look like this:

"dependency-search_paths": ["lib"],
"targets": {
	"game-release": {
		"type": "executable",
		"target": "macos-aarch64",
		"sources": ["game/src/**", "src/**"],
		"dependencies": ["lib_for_engine", "lib_for_platform"],
	},
}

As this is a manual process, it is prone to error and the complexity grows exponentially as more libs or target configurations are added.

2. Build each target individually

This allows each target to manage its own dependencies and is much easier to maintain.

The downside is that you now have to manually build each target in order, which is usually solved by writing project-specific build scripts and can also be a burden to maintain across multiple platforms.

Proposed solution

Provide a new field, target-dependencies, to project.json that will build each target based on it's definition:

"dependency-search_paths": ["lib"],
"targets": {
	"game-release": {
		"type": "executable",
		"target": "macos-aarch64",
		"sources": ["game/src/**"],
		"target-dependencies": [
			"platform",
			"engine"
		],
	},
		"engine": {
		"type": "static-lib",
		"target": "macos-aarch64",
		"sources": ["src/engine/**"],
		"dependencies": ["lib_for_engine"],
	},
		"platform": {
		"type": "static-lib",
		"target": "macos-aarch64",
		"sources": ["src/platform/**"],
		"dependencies": ["lib_for_platform"],
	},
}

Constraints

• Must build targets in the order they are declared in the target-dependencies field.
• Must detect and prevent recursive dependencies, and provide feedback to the user in such cases.
• Should provide feedback to the user when a target dependency targets a different architecture to the calling target.
• Won't provide any configuration inheritance between targets and dependencies (out of scope).

[lerno]

There are two ways of doing this:

1. The compiler compiles each project in order.
2. The compiler shells out and invokes itself with the other targets.

There are some effects to each. Presumably invoking a target implicitly as a dependency should not be affected by the particular settings passed on the command line. However, there might be cases where this is desired? In any case (2) solves this problem, but some build options might be desirable to pass on, such as where everything is running? (1) could also solve this, but there would need to be a bit more discipline involved.

Picking (2) has disadvantages as well, such the compiler needs to know what is already resolved.

For example, imagine we have 5 targets: A, B, C, D and E. B depends on A, C depends on A and D, E depends on B and C. If we shell out to build B, this in turn shells out to build A. Then C is to be built, but we don't want A to be built as it was recently built. The solution is either to use files to track when the targets are built, or pass on an additional build option. For example:

c3c build d
-> calls
  c3c build b
  -> calls 
    c3c build a 
-> calls
  c3c build c -nobuild b -nobuild a
  -> calls
    c3c build d -nobuild b -nobuild a

Alternatively, the dependencies are completely worked out like:

c3c build d
-> calls
  c3c build a -nodep
  c3c build b -nodep
  c3c build d -nodep
  c3c build c -nodep

Of course, if (1) is used then this latter version will implicitly be used.

[terids]

It seems much easier to reason about if all target-dependencies are parsed first to build a list of unique targets and the order they would need to be built to satisfy the dependency chain. No files needed for tracking in this case. This also gives the build system a chance to detect any invalid configuration and abort early before any compilation work has started.

Some pseudo-code of how I imagine it:

Build B:
	build_target_list
	::Run
		build_target_list = GatherDependencies(B)
		Validate(build_target_list)

		build_target_list.reverse()
		Foreach target in build_target_list
			exec build_single_target(target)

	::GatherDependencies(target) -> Return list
		list

		Foreach target_dependency in target
			list.AddUnique(GatherDependencies(target_dependency))
		
		list.AddUnique(target)

		return list

	::Validate(targets)
		Foreach index, target in targets
			if index != targets.lastindex && target == targets.last
				Abort("Error: Cyclic dependency detected. <target> cannot depend on <targets.last>")
		// Any other validation rules here

Examples

// Success case
Target: Dependencies
	A: 
	B: A C D
	C: A D
	D:
	E: B C
// Build B - Produces a build target list of: A D C B

// Fail case
Targets:
	A: 
	B: A E C D
	C: A D
	D:
	E: B C
// Build B - Produces "Error: Cyclic dependency detected. E cannot depend on B"

One other thing to consider; Does it always make sense to rebuild a target?

For example, if A and D are static libs, should they be rebuilt every time the executable target is built? If it wasn't going to be rebuilt, passing command line arguments might become confusing if passing something like "-LOGGING" that would affect the compiled code, in which case it's better that dependencies aren't affected by the command line arguments and these should be set in the target.json for that target.

If it was possible to add a source-only target type it would make more sense to pass command line args along, but I don't think it's possible to split source up like this at the moment?

[lerno]

What do you mean by a source-only target type?

[terids]

A target that doesn’t compile to anything on its own, but can be added as a dependency to be built as part of an executable or static-lib/dynamic-lib/object-files.

I’m probably thinking along the lines or a sub-target that lets you define the build configuration/dependencies for that part of the codebase.

[lerno]

Those already exist, you set the target type to "prepare".

[terids]

A .c3l library is more what I was thinking, without having to specifically create a separate library and move source files. The c3l system already covers this use case with manifests, so my source-only comment can be considered moot.

Should a dependency always be built?

It makes more sense for some than others. For the table below, I would consider outdated to be if any source file modified timestamp is more recent than the output artefact timestamp, or if that target's configuration changed since last built. Are there any other 'outdated' cases I'm missing?

Target Type	Always build?
executable	only if outdated
static-lib	only if outdated
dynamic-lib	only if outdated
benchmark	yes
test	yes
object-files	only if outdated
prepare	yes

This raises the complexity of the task quite considerably, and makes it harder to communicate to the user when a dependency would be built.

It could be left up to the user by adding another field:

"rebuild-when-dependency": [] // "always", "outdated", "never" (default: "always")

This feels beyond the scope of this issue.

Should command line arguments be passed to target dependencies?

I'm in favour of not passing command line arguments as it is not possible to reason about which targets should consume which arguments.

As one of the benefits of implementing target dependencies is to reduce the need to create separate build scripts, it would make sense to let the user define arguments on individual targets in project.json as this gives most flexibility.

"args": [] // A list of arguments to pass to c3c build for this target

This leans towards option 2 of having the compiler invoke itself with other targets, but I don't have any knowledge of how this is currently written so wouldn't want to give bad advice.