Allow packages to control what goes into the build hash

[calebkiage]

Is your feature request related to a problem? Please describe.

Right now, a package's buildhash is computed using a fixed set of properties. This is fine for libraries and binaries, but not for scripts for example, or packages that contain only data.

Describe the solution you'd like

The suggestion here is to provide a way for packages to update what they would like to include in the buildhash computation.

Describe alternatives you've considered

None

Additional context

I've started exploring an initial PoC here https://github.com/calebkiage/xmake/tree/feat/package_abi_feature

package("noop")
    add_configs("trivial", {description = "A non-abi changing option"})
    add_configs("changing", {description = "An abi changing option"})

    add_deps("foo")

    on_load(function (package)
        -- Clears ABI defaults
        -- package:clear_abi()
        -- adds a config to the hash calculation
        package:set_abi("config.changing")
        -- Updates version ABI to use 1.3.x
        package:set_abi("version", package:version():major() .. (package:version():minor() and ('.' .. package:version():minor()) or ''))
        package:add_abi("deps", "config.shared", "config.debug")
    end)

    on_install(function (package)
    end)

[waruqi]

    package:set_abi("config.changing")
   package:set_abi("version", package:version():major() .. (package:version():minor() and ('.' .. package:version():minor()) or ''))
   package:add_abi("deps", "config.shared", "config.debug")

The buildhash already contains the configs and version changes, so you don't need to do anything extra.

For now, I'm not considering exposing the API to control buildhash, it's too much of a hack.

[calebkiage]

This is what other successful package managers like Conan do right now. How is it a hack? I don't understand. It's still early and I set this up in less than a day but the need is there and design can be iterated on

[waruqi]

Maybe I'll implement it in the future (when I am going to do the linux/macos pre-compiled packages), but for now I haven't figured out the API design.

You can provide some API design suggestions here that I might take up in the future, or implement a better API based on the discussions here, but I probably wouldn't consider accepting pr for now.

[calebkiage]

Maybe we can start out with what you think is wrong with the API and we can go from there. As it stands, I don't know what to address.

[waruqi]

I'm not saying that the current API is wrong, I just haven't had the time to think about how to design the API, which may require developing actual requirements to continually improve the API and validate it. (e.g. support for pre-compiled binary packages).

I prefer to automate the handling of package buildhash and ABI rather than letting the user modify the settings for it, which would raise the complexity of package maintenance.

For example, it might be possible to improve the toolchain module to generate the abi hash and then bind it to the package, etc.

Sorry, my free time is fragmented these days and it's hard to settle down and spend time thinking about whether some new API design makes sense.

[calebkiage]

I think using the toolchain is an interesting idea. but that means every package needs to use a toolchain. Some packages might not get compiled at all so they won't have a toolchain.

[waruqi]

I think using the toolchain is an interesting idea. but that means every package needs to use a toolchain. Some packages might not get compiled at all so they won't have a toolchain.

This is the point where it takes time to think about and design the API. Maybe in the future, I'll refactor some of the packages and the overall design of toolchain, which may bring some of the more significant changes.

This feature requires a lot more changes than just a set_abi. I need to spend a lot of time thinking about how to implement it better.

[calebkiage]

It's good that this is something you might be interested in. I know the final feature might be a lot bigger than just the few things I put together in that PR. I'm willing to work with you to bring this feature out even better.

As I was working on this, I also thought about using dependencies' buildhash to determine the resulting hash and that could be opted into by the package. For example, a header only library doesn't depend on its dependencies' ABI, but a shared library does. A package author can say that package-a doesn't depend on a header library dependency's buildhash by saying:

-- in package("header-only-dep")
package:remove_abi("config.shared") -- This package will not change the build hash if shared changes
package:remove_abi("config.debug") -- This package will not change the build hash if debug changes
package:remove_abi("compiler") -- This package will not change the build hash if the compiler changes
package:remove_abi("system") -- This package will not change the build hash if the platform changes
-- The above could all be defaulted if package kind is header only.

-- in package("a")
package:add("deps", "header-only-dep")
package:set_abi("deps") -- Adds all deps to ABI calculation (their buildhash is included)

When I change my compiler, the header only dependency doesn't change. Only package a is rebuilt

[calebkiage]

Thinking about it a little more, I don't think the package build hash should be in the toolchain.

Maybe the toolchain will have its own hash and that hash will be used in the package's build hash as compiler. This would mean making the toolchain available as a parameter passed into the package when creating it. Right now, it's not very easy to find out what toolchain was used to create a package. The existing build hash uses a parameter that is sometimes empty. I think as part of this, I can add the feature to pass in a toolchain into the package so that package:toolchain() always has a value.

Another useful supporting feature is having package:deps() always available...I'll have to think about that some more.