Entirely generated configfiles?

[nwf]

Is your feature request related to a problem? Please describe.

I find myself wanting to generate some configfiles entirely in Lua. At the moment, I am doing this with a series of single-line configfiles each mentioning single variables (${thing1}, ${thing2}, and so on) and having the target use those:

target("foo")
	set_kind("phony")
	set_default(false)

	add_imports("core.base.json")

	on_load(function (target)
		import("core.project.config")
		-- compute some things from config and by reading some JSON files
		-- and store the results with target:set().
	end)

	add_configfiles("thing1.in", { filename = "thing1" })
	add_configfiles("thing2.in", { filename = "thing2" })

	on_config(function(target)
		-- compute some things from target:get("...") properties

		target:set("configvar", "thing1", --[[ ... --]])
		target:set("configvar", "thing2", --[[ ... --]])
	end)

All the state tracking in maybe_copyfiles is really nice, and later stages rerunning their build and link steps based on changes in config files just works.

Describe the solution you'd like

I'd prefer to use a callback with add_configfiles that didn't need a source file and had access to the target

add_configfiles(nil, { filename = "thing1",
  generator = function(target) return target:get(--[[ ... --]]) end
})

Describe alternatives you've considered

No response

Additional context

No response

[nwf]

I guess another approach is "one configfile per target" and have each target's on_config() run target:set("configvar", "thevar", --[[ ... --]]) and use a single template file that's just ${thevar}.

[waruqi]

I still don't understand what you want to do, is there something wrong with add_configfiles currently?

[nwf]

AFAIK, add_configfiles always takes an input file. I'm not sure that's "wrong", but it'd be nice if it could just manage writing a string to a file, so that I didn't need things like https://github.com/CHERIoT-Platform/cheriot-rtos/pull/537/commits/63936fbdaa844b71681080e0882e008ee9e646bc:

-- Write contents to path if it would create or update the contents
local function maybe_writefile(xmake_io, xmake_try, path, contents)
	xmake_try
	{ function()
		-- Try reading the file and comparing
		local old_contents = xmake_io.readfile(path)
		if old_contents == contents then return end
		xmake_io.writefile(path, contents)
	  end
	, { catch = function()
		-- If that threw an exception, just write the file
		xmake_io.writefile(path, contents)
	  end
	} }
end

which gets used as in https://github.com/CHERIoT-Platform/cheriot-rtos/pull/537/commits/c7348543724d1f39c1157ba909011406fb9d4d83 like this:

	on_build(function(target)
		-- ...
		maybe_writefile(io, try, target:targetfile(),
			format("__mmio_region_start = 0x%x;\n%s__mmio_region_end = 0x%x;\n__export_mem_heap_end = 0x%x;\n",
				mmio_start, mmio, mmio_end, board.heap["end"]))

because add_configfiles already manages similar feats internally: https://github.com/xmake-io/xmake/blob/7301a531f1581e5b22897c13c66d2704c493dbd9/xmake/actions/config/configfiles.lua#L360-L377

[waruqi]

because already manages similar feats internally:add_configfiles

I still don't understand what you need now. If you need configfiles to be written only when the file content changes, doesn't add_configfiles already do something similar internally? What are the other problems?

[nwf]

AFAIK, configfiles always come from another file (usually named something .in): https://github.com/xmake-io/xmake/blob/7301a531f1581e5b22897c13c66d2704c493dbd9/xmake/actions/config/configfiles.lua#L297 which is then modified by variable substitutions in various forms. (These transformations do not have access to much build system state: a configvar and, perhaps, a preprocessor and arguments provided by the input file.)

The example I gave above does not have a .in source file and is not the result of substituting configvars into a larger template. Instead, the output file contents are entirely generated in Lua. I cannot use the configfile machinery to achieve this outcome unless I have a .in for each generated file (each containing a single variable) or I have a shared .in (again, containing a single variable) and a target() per generated file. This is slightly unfortunate because maybe_writefile duplicates functionality already present in the configfile machinery and, also, is a little obnoxious to use, needing try and io passed to it in addition to the "obvious" arguments.

[waruqi]

Why can't we do it with config.h.in and configvars? I thought it could do anything, even generating from a string as you said.

config.h.in

${CONTENT}

target("xxx")
    add_configfiles("config.h.in")
    on_load(function(target)
        local vars = {foo = 1, bar = 2}

        -- you can generate all content to content string
        local content = generate_content_with_allvars(vars) -- TODO
        target:set("configvar", "CONTENT", content)
    end)

[nwf]

Why can't we do it with config.h.in and configvars? I thought it could do anything, even generating from a string as you said.

Yes, that's the "have a shared .in (again, containing a single variable) and a target() per generated file" approach.

ETA: Another downside of that approach, relative to maybe_writefile is that all configfiles are generated, even if their containing target isn't depended upon by the built root targets.

[waruqi]

ETA: Another downside of that approach, relative to is that all configfiles are generated, even if their containing isn't depended upon by the built root targets.maybe_writefiletarget

So, this is just an issue of configfiles being generated globally. It only needs to be improved to support generating configfiles only for the targets that need to be built.

https://github.com/xmake-io/xmake/blob/d558ace9f0d9daa3c51d3ac6b56147a67e995149/xmake/actions/config/configfiles.lua#L33