Add an ability to merge fileconfigs in description scope

[GooRoo]

Is your feature request related to a problem? Please describe.

Since all our projects are usually similar in terms of building process and folder structure, I'm trying to simplify the way of bootstrapping new projects. For this, I'm trying to come up with some clean DSL.

For example, instead of writing

target('myapp')
    add_rules('qt.quickapp')
    add_rules('@easy/qrc', '@easy/qmldir')  -- rules for autogeneration
    add_files('src/**.cpp')

    add_extrafiles('qml/**', {
        easy_qrc = true, 
        qrc_prefix = '/qt/qml', 
        qrc_base_dir = 'qml', 
        easy_qmldir = true,
        qmldir_module = 'mega',
		qmldir_version = '1.0',
		qmldir_prefer = ':/qt/qml/mega/'
		-- and so on...
    })

I'd prefer to write something like this:

target('myapp')
    add_rules('qt.quickapp')
    add_rules('@easy/qrc', '@easy/qmldir')  -- rules for autogeneration
    add_files('src/**.cpp')

    add_qt_resources('qml/**', {prefix = '/qt/qml', base_dir = 'qml'})
    
    generate_qmldir_for('qml/**', {module = 'mega', version = '1.0', prefer = ':/qt/qml/mega/'})

It's cleaner, easier to understand, to write and to teach somebody.

The problem is with these two custom functions, that may looks like this:

function add_qt_resources(...)
    -- some pre-checks and transformations
    -- ...

    add_extrafiles('qml/**', extra_opts)
end

function generate_qmldir_for(...)
    -- ...

    add_extrafiles('qml/**', extraopts)
end

The problem with this approach is that the second call to add_extrafiles on the same paths overrides the fileconfigs for the files.

I need a way to merge the fileconfigs in the description scope (since I'd like to keep everything declarative, i.e., without any additional scripts).

Describe the solution you'd like

I see several possibilities how to give an ability to merge the configs.

1. merge = true

The first one could looks like this:

add_extrafiles('qml/**', {merge = true, easy_qrc = true, qrc_prefix = '/qt/qml'}) 
add_extrafiles('qml/**', {merge = true, easy_qmldir = true, qmldir_module = 'mega'})

or

add_files('qml/**', {merge = true, rules = '@easy/qrc', qrc_prefix = '/qt/qml'}) 
add_files('qml/**', {merge = true, rules = '@easy/qmldir', qmldir_module = 'mega'})

So, merge = true would regulate the behavior. The problem with this one is that xmake would need to know which keys should be merged into a list and which should be simply replaced (if in both calls there identicall keys). For example, the rules should become {'@easy/qrc', '@easy/qmldir'}, but for other options the rule would expect to get a string, not a table.

2. rules, add_rules + merge = true

This approach is similar to the previous one except the fact that merge = true would simply do table.join() effectively overriding overlapping keys.

But for rules, there could be a separate option – add_rules which does the same as rules, but would merge the values:

add_files('**', {add_rules = {'foo', 'bar'}, a = 1, b = 2})
add_files('**', {add_rules = {'buzz'}, b = 3, c = 4, merge = true})
-- should result in {rules = {'foo', 'bar', 'buzz'}, a = 1, b = 3, c = 4}

3. Introduce tags (preferred)

The actual concern is how to pass lists of various files from the target into the rule with additional parameters (see also #6723, #6728). So, instead of using existing add_files (and marking them with particular rules manually) or using add_extrafiles, a new API can be introduced: mark any files with tags (with parameters) and allow rules to work with them.

For example:

target('myapp')
    add_rules('qt.quickapp')
    add_rules('easy-qrc', 'easy-qmldir')  -- rules for autogeneration
    add_files('src/**.cpp')

    add_file_tags('qml/**', 'easy.qrc', {prefix = '/qt/qml', base_dir = 'qml'})
    add_file_tags('qml/**', 'easy.qmldir', {module = 'mega', version = '1.0', prefer = ':/qt/qml/mega/'})

rule('easy-qrc')
    set_tags('easy.qrc')
    on_config(function(target)
        local files, fileconfigs = target:tags('easy.qrc')
        -- some processing
    end)

rule('easy-qmldir')
    set_tags('easy.qmldir')
    on_config(function(target)
        local files, fileconfigs = target:tags('easy.qmldir')
        -- some processing
    end)

The pros are that the new API could allow the same files to be tagged with different tags at the same time and that the fileconfigs are scoped to tags (e.g., prefix for easy.qrc tag and prefix for easy.qmldir tag would be independent).

Describe alternatives you've considered

No response

Additional context

No response

[definitelythehuman]

Wow, sounds very nice and convenient! Adding tags will allow to process files with same extension by separate rules!

[GooRoo]

@definitelythehuman This is already possible. But now, I cannot tell xmake to process the same files with two different rules via two separate calls in the target, where each of them sets its own parameters like this:

target('myapp')
    add_rules('first', 'second')
    add_extrafiles('qml/**', {foo = 1, bar = 2})
    add_extrafiles('qml/**', {bar = 3, buzz = 4})

because the resulting fileconfig will be {bar = 3, buzz = 4}, not the combination of all of them.

[waruqi]

we can use new apis to override and merge them. However, it will be very complicated to implement and cannot be supported in a short time.

set_extraconfs("files", "src/*.cc", {foo = 1, bar = 2}) -- override
add_extraconfs("files", "src/*.cc", {foo = 1, bar = 2}) -- merge

add_extraconfs("extrafiles", "qml/*.qml", {foo = 1, bar = 2})

[GooRoo]

@waruqi The approach with tags may also be used to pass certain artifacts between rules.

For example, I have a rule that compiles shaders. One user may decide to put the compiled results next to the target binary on the file system. Another user may want to pack them into Qt resources using my qrc generator rule.

How exactly can a user organize this on the target level easily? (See the discussion on discord).

With tags, the rule that compiles shaders may “tag” its output artifacts with, let's say, "shader.compiler.output". And a user can then re-tag them with "easy.qrc" like this:

target('myapp')
    add_rules('qt.quickapp')
    add_files('src/**.cpp')

    add_rules('shader-compiler')  -- the rule to compile shaders
    add_rules('easy-qrc')  -- the rule to generate QRC for packing into resources

    -- now tag the outputs of one rule as the input of another
    add_tags('shader.compiler.output', 'easy.qrc', {prefix = '/'})

[waruqi]

why not use target:data("shader.compiler.output"), target:data_set("shader.compiler.output", "xxx") to pass them between rules/targets?

https://github.com/xmake-io/xmake/blob/7c08a859433be31e67aa4ef4a7968060a1e57ca5/xmake/rules/wdk/mof/xmake.lua#L79

and set_values/add_values/target:values/target:values_set can also do these in user configuraion side.

[GooRoo]

I don't think target:data_set is a good idea, since ideally I'd like my rules to be unaware of each other. But I'll try to utilize set_values for this purpose.