[vcpkg-tool-llvm] more tool ports

[Neumann-A]

• [x] depends #23911 for the policiy setting

[JackBoosY]

baseline issue?

[Neumann-A]

baseline issue?

yes

[JackBoosY]

One regression last.

[Neumann-A]

One regression last.

According to the observations in other PRs this seems to be a baseline issue.

[JackBoosY]

@Neumann-A Yes, I'm fixing this.

[JackBoosY]

Depends on https://github.com/microsoft/vcpkg/pull/23999

[strega-nil-ms]

marking requires:discussion for team discussion - I don't understand why this is important vs. vcpkg_find_acquire_program?

[Neumann-A]

I don't understand why this is important vs. vcpkg_find_acquire_program

I thought you wanted to get rid of vcpkg_find_acquire_program? Also vcpkg_find_acquire_program does not support versioning. So you are stuck with whichever clang is currently in vcpkg_find_acquire_program

[Neumann-A]

@strega-nil-ms: Results? I need this to push out the usage of yasm.

[JackBoosY]

cc @BillyONeal

[JackBoosY]

@BillyONeal I think we have the conclusion now: we can accept these tools, so we don't need any discussion right?

[BillyONeal]

@BillyONeal I think we have the conclusion now: we can accept these tools, so we don't need any discussion right?

I think we want to answer what this means with respect to artifacts. Artifacts are the tool we really want to solve this problem but they're nowhere near stable enough to build the vcpkg tree on top of yet ...

[Neumann-A]

I think we want to answer what this means with respect to artifacts. Artifacts are the tool we really want to solve this problem but they're nowhere near stable enough to build the vcpkg tree on top of yet ...

Probably just update of the dependent ports? You will probably have a json field artifacts: to give vcpkg information which artifacts are required for a certain port. In the end it doesn't matter if the tools are provided by artifacts, ports or vcpkg_find_acquire_program (allthough the last does not play well with versioning).

[BillyONeal]

After discussion with @ras0219-msft, @dan-shaw, and @vicroms today, we have the following notes/comments:

• vcpkg_find_acquire_program will be, over time, changed into equivalent vcpkg fetch calls. This will fix bugs where vcpkg_find_acquire_program was not respecting things like VCPKG_FORCE_SYSTEM_BINARIES and friends. Example: https://github.com/microsoft/vcpkg/issues/21482
• vcpkg_find_acquire_program and fetch model "the canonical system copy" of whatever they expose, since those bits are often intended to come from the system. This means that they may not be version constrained, aren't intended to participate in the ABI hash, and similar.
• The maintainers agree that moving to a more "declarative" mechanism rather than vcpkg_find_acquire_program would be an improvement, but aren't sure if a stop-gap before we have the full artifacts-driven solution available would be enough of an improvement to offset adding a 4th exposed way to get at what fetch morally does. If you still want to make progress here, if we could commit to adding vcpkg-BIKESHED-foo replacing all vcpkg_find_acquire_program calls with that in the tree in a reasonable timeframe, that may be OK. (Where BIKESHED is something like "system" or "find-acquire-program" or similar).
• We still do not want vcpkg ports to become a source of tools, and want to discourage providing things in $installed/tools wherever possible. That means we probably do not want vcpkg-tool-7zip or vcpkg-tool-llvm. An LLVM artifact may make sense, but we do not have a committed time where that will be available other than to say "we are working on it". Getting vcpkg artifacts to be able to deploy compilers to use, and that be used as, effectively, the "triplet toolset" is a problem we do need to solve.

For Python specifically:

• For software that just wants a Python interpreter to run a python script, like Meson, and isn't going to be loading into python as a development tool / plugin, etc. they should continue depending on the system copy of Python, as today.
• If a library does find_package(Python COMPONENTS Interpreter), it may make sense to use the "system" copy of Python to resolve this dependency, if and only if that selected copy of Python is not exposed to exported CMake configs.
• If a library does find_package(Python anything-here-that-needs-more-than-an-interpreter) it must build Python from source by depending on the Python port.
• Ports in general need to understand this difference between the system installed Python and a built-from-source Python. Situations where "all versions must be equal" need to use the from-source copy.

See also:

• https://github.com/microsoft/vcpkg-tool/pull/575
• https://github.com/microsoft/vcpkg-tool/pull/577

[Neumann-A]

@BillyONeal I cannot really read a plan of action out of your statement.

This means that they may not be version constrained, aren't intended to participate in the ABI hash, and similar.

I understand that for 7Zip but in light of #23660 and #24622 some tools need to participate in ABI hashing somehow. Furthermore vcpkg_find_acquire_program currently is hashed so I don't really count that arguments currently until that is changed somehow.

if we could commit to adding

Is the if a typo there? Could the team commit to something like that?

BIKESHED

I would opt for BIKESHED being tool. I don't see a reason to name it system or find_acquire_program. It is for the portfile (see msmpi/cuda/mkl etc.) to decide how it plans to fulfill a dependency and vcpkg-tool-foo is as descriptive as it gets in that regards. The portfile would probably just do what vcpkg_find_acquire_program does but within the vcpkg-port-config.cmake instead of the portfile.cmake.

We still do not want vcpkg ports to become a source of tools, and want to discourage

I understand that but I need a solution not a we are working on it with a unknown time frame. So would it be ok to replace the current behavior in vcpkg-tool-7zip, vcpkg-tool-llvm with something which has a similar behavior within the vcpkg-port-config.cmake instead of the portfile.cmake? This would effectively mimic the behavior of vcpkg_find_acquire_program.cmake and just move it into vcpkg-port-config.cmake. It has the benefit that the dependency is made explicit and version changes are reflected. The benefit how having ports for it you could add a field abi-relevant (defaults to true) to the ports manifest to deactivate the hashing of certain ports. Furthermore you could add a field (provided-)artifact| to work together with vcpkg-ce or vcpkg fetch. It also has the benefit if it is an unknown host where neither artifacts or a binary release is available it can fallback to building from source or the behavior can easily be changed by providing an overlay for it.

Slightly OT

Ports in general need to understand this difference between the system installed Python and a built-from-source Python.

Tell that the meson guys. I just recently needed to patch a port using python.dependency() or something similar. People upstream are hardcoding interpreter = libraries.

Furthermore python scripts like meson can have a minimum python version requirement which is not reflected by the current handling within vcpkg. We should deal with the python + meson situation just like cmake where we force the (minimum) version required. meson requires 3.7 which seems currently be ok with vcpkg but what is with other scripts which might require 3.10 ? (I don't want to discuss this in: I just want to raise the issue here since I feel like that has been forgotten.) /OT

[Neumann-A]

@BillyONeal So I moved the whole tool stuff into vcpkg-port-config and let it make a find_program call. If it finds nothing it tries to acquire it. So feel free to discuss if this is what you want until the team someday has artifacts going.

[JackBoosY]

Ping @BillyONeal for reply.

[eli-schwartz]

Slightly OT

Ports in general need to understand this difference between the system installed Python and a built-from-source Python.

Tell that the meson guys. I just recently needed to patch a port using python.dependency() or something similar. People upstream are hardcoding interpreter = libraries.

Furthermore python scripts like meson can have a minimum python version requirement which is not reflected by the current handling within vcpkg. We should deal with the python + meson situation just like cmake where we force the (minimum) version required. meson requires 3.7 which seems currently be ok with vcpkg but what is with other scripts which might require 3.10 ? (I don't want to discuss this in: I just want to raise the issue here since I feel like that has been forgotten.) /OT

The minimum python version is actually very relevant for Meson, too, as you pointed out here: https://github.com/microsoft/vcpkg/issues/27117#issuecomment-1270515887 People are running vcpkg on Ubuntu 18.04 which ships with a system python 3.6 (with pip install meson that just downloads an old version of Meson that runs fine, so you can cross your fingers and hope the project you are building doesn't require a newer version of Meson than the last one to support 3.6).

The python interpreter to build compiled CPython extensions can be solved by using meson machine files to point the python = "binaries" entry to the one that vcpkg installs as a non-tool.

[BillyONeal]

Under the current tools proposal I am in favor of vcpkg-tool-llvm but not vcpkg-tool-7zip. Will let you know after disucssion.