Build Command with Multiple Inputs and Multiple Outputs

[tgockel]

What are you trying to do?

I am trying to use a Premake custom build command for a program which generates code that takes multiple inputs and creates multiple outputs. However, I can't figure out the Premake to take multiple inputs and generate multiple outputs.

What have you tried so far?

The Protocol Buffer compiler protoc is one such program that operates in this way that I will use for demonstration. This script works for taking all of the .proto files and compiling them individually:

workspace "CustomBuildCommandTesting"
    configurations { "Debug", "Release" }
    location "build"

project "custom-buildcommand-test"
    kind "ConsoleApp"
    language "C++"
    targetdir "build/%{cfg.buildcfg}/bin"

    files { "src/**.cpp", "src/**.proto" }
    includedirs { "%{cfg.objdir}/generated" }

    -- Using protoc for demo, but the question applies to anything with multiple inputs
    filter 'files:**.proto'
        buildcommands {
            'mkdir -p "%{cfg.objdir}/generated"',
            'protoc --proto_path=../src --cpp_out="%{cfg.objdir}/generated" "%{file.relpath}"',
        }

        buildoutputs {
            '%{cfg.objdir}/generated/%{file.basename}.pb.h',
            '%{cfg.objdir}/generated/%{file.basename}.pb.cc',
        }

This creates rules in Make that look like this (cleaned a bit for clarity):

obj/Debug/generated/foo.pb.h: ../src/foo.proto
    mkdir -p "obj/Release/generated"
	protoc --proto_path=../src --cpp_out="obj/Release/generated" "../src/foo.proto"

obj/Debug/generated/bar.pb.h: ../src/bar.proto
	mkdir -p "obj/Debug/generated"
	protoc --proto_path=../src --cpp_out="obj/Debug/generated" "../src/bar.proto"

As an aside, there is no mention of the .pb.cc files that get generated anywhere, which is not ideal. It seems like that should get picked up from buildoutputs, but I must be missing how to make that happen.

What problem are you having?

The real problem here is this isn't the way you're supposed to use protoc. For protoc, you want to use all of the inputs to generate all of the outputs in one pass. A more-correct rule in Make would look like this:

obj/Debug/generated/bar.pb.h  \
obj/Debug/generated/bar.pb.cc \
obj/Debug/generated/foo.pb.h  \
obj/Debug/generated/foo.pb.cc \
        &:                    \
        ../src/bar.proto      \
        ../src/foo.proto
	mkdir -p "obj/Debug/generated"
	protoc --proto_path=../src --cpp_out="obj/Debug/generated" $^

How do you write a Premake command that can take multiple inputs and generate multiple outputs? If that isn't possible with custom commands, then what mechanism is available in Premake?

What version of Premake are you using?

premake5 (Premake Build Script Generator) 5.0.0-dev (it's from the beta 2 binary)

Anything else we should know?

For completeness, here's the full demo source tree.

.
├── premake5.lua
└── src
    ├── bar.proto
    ├── foo.proto
    └── main.cpp

src/foo.proto:

syntax = "proto3";

message Foo {
    uint64 value = 1;
}

src/bar.proto:

syntax = "proto3";

import "foo.proto";

message Bar {
    Foo foo = 1;
}

src/main.cpp:

#include "bar.pb.h"

int main()
{
}

[Jarod42]

As an aside, there is no mention of the .pb.cc files that get generated anywhere

Some generators (as gmake* ) only considers first buildoutputs in its rules. I suggest to place *.pb.cc first then

Note: compilebuildoutputs "on" seems missing to compile the generated *.pb.cc afterward.

A more-correct rule in Make [..]

In the same way, compiler might compile several sources at once, rule created only handle one source at a time. There are no ways to explicitly have that rule.

For proto, the rule seems correct though (even if not idiomatic?). Suppose only "bar.proto" is modified, Would we want to regenerate other files too?

what mechanism is available in Premake?

Other option seems worst in your case:

• prebuildcommand which is unconditional
• Create another project with kind "Makefile"
• CustomRule similar to custom command (no supported by gmake (but ok with gmake2), xcode)

[tgockel]

My workaround here is to scan the proto_files and build up the outputs list by hand (easy enough here, since {base_dir}/file.proto -> {gen_dir}/file.pb.h and {gen_dir}/file.pb.cc), then pick a single proto file to use as the argument to filter, like so:

    -- just pick _a_ proto file -- it doesn't matter which
    filter('files:'..proto_files[1])
        buildinputs(proto_files)
        buildoutputs(outputs)

        buildcommands {
            'mkdir -p "%{cfg.objdir}/generated"',
            'protoc --proto_path="'..src_basedir..'" --cpp_out="%{cfg.objdir}/generated"'..proto_files_arg,
        }

        compilebuildoutputs "on"
    filter{}

This gets these rules in Make:

obj/Debug/generated/foo.pb.h: ../src/foo.proto ../src/foo.proto ../src/bar.proto
    mkdir -p "obj/Debug/generated"
    protoc --proto_path="../src" --cpp_out="obj/Debug/generated" "../src/foo.proto" "../src/bar.proto"

obj/Debug/generated/foo.pb.cc obj/Debug/generated/bar.pb.h obj/Debug/generated/bar.pb.cc: obj/Debug/generated/foo.pb.h

It works well enough for me. Just a note, the proper way to handle "compiler might compile several sources at once" in Make is "Rules with Grouped Targets": you basically use &: instead of : and Make understands that the targets will all be generated from the recipe.

Suppose only "bar.proto" is modified, Would we want to regenerate other files too?

In Proto, the answer is "yes." One could have an incremental build system, where you detect all the transitive imports like you have to do with #includes in C land, but running the protoc codegen is fast enough that it doesn't really matter. Unfortunately, protoc will always update even when there are no changes, but that issue is also workaroundable. Being smarter on this front has been deemed "not worth the complexity" by the Protobuf devs.

[Jarod42]

then pick a single proto file

I uses similar thing for a n-inputs 1-output rule.

[Jarod42]

Question has been asked/copied to https://stackoverflow.com/questions/77490364/premake-build-command-with-multiple-inputs-and-multiple-outputs