Adding `host_exports` (like `run_exports` except between `build` & `host` only)

[jakirkham]

run_exports has been immensely helpful. Just to summarize it run_exports:

• Allows packages with knowledge of what they need to express it for easy downstream consumption
  • Packages that are part of the build can add dependency to run
    • weak only happens when the dependency is added to host (not build)
    • strong happens when a dependency is added to build or host
  • Dependencies added can express ABI compatibility

However there are still some cases that are not covered. In particular there are cases where a dependency added to build needs to add itself to host, but there is no need to add it to run (in fact this would often be better to avoid). IOW run_exports/strong does not meet this need. Some examples include:

• Header-only dependencies of a package
  • https://github.com/conda/conda-build/issues/3796#issuecomment-549431589
• Tighter alignment in build & host of compiler dependencies (like OpenMP)
  • https://github.com/conda-forge/openmp-feedstock/issues/126
• Compiler runtime libraries that are statically linked (like cudart)
  • https://github.com/conda-forge/staged-recipes/pull/21723#issuecomment-1405444862
• Other compiler use cases
  • https://github.com/conda/conda-build/issues/3966

Solving these cases would involve a host_exports that allows adding other dependencies to host. Could imagine to use cases for a package providing host_exports

• host_exports/weak (default)
  • Add dependencies to host if providing package is added to host
  • Solves the header-only dependency case above for a library
• host_exports/strong (opt-in)
  • Add dependencies to host if providing package is in build or host
  • Handles the compiler dependency alignment problem (with OpenMP)
  • Handles statically linked compiler libraries (like cudart)
  • Of most use for compiler use cases

[jakirkham]

cc @xhochy (for awareness)

[jakirkham]

@jaimergp @wolfv would be interested to hear your perspectives on this idea? 🙂

[jaimergp]

I'm generally supportive of the idea provided that it's useful for a wide range of cases. A quick look made me see it as a "couple of edge cases". For example, it's not clear to me what "Add dependencies to host if providing package is added to host" means; if the package is already in host, why do we need to export anything? Is it to constrain it further? I guess we can use pin_compatible but we want to avoid that "annoyance" if the package in build already knows how tell host, right?

[jakirkham]

Thanks Jaime! 🙏

Am happy with limiting it to the case where a package is added to requirements/build

So that would mean there is no need for host_exports/weak and host_exports/strong could just be host_exports

Does that same more reasonable?

[jaimergp]

run_exports also started as a flat value with just the list of specs and then later decided to allow the map form with weak and strong, so I could see how we start with just host_exports: [specs]. Anyway, I'm not the most knowledgeable person here to dictate how this design should look like, but I do want to help you get the ball rolling. I think what you have already is good enough for a CEP PR and then we all can discuss the different options there!

[h-vetinari]

[I started drafting this a long time ago; it quickly got quite extensive for a comment and I was wondering if I should take it to the next level right away, i.e. a CEP draft. In the end I decided to gauge the appetite for something like this before I write a CEP]

This need also comes up in dealing with C++ resp. Fortran modules, which are dependent on the ABI of the specific compiler that produced them. There are various ways we could attach a _fortran_modules_abi constraint to a foo-devel package that contains Fortran modules (e.g. produced by flang, and we want to enforce that the fortran compiler in a dependent package needs to match).

The problem is that in such a scenario

  - name: i-consume-fortran-modules
    requirements:
      build:
        - {{ stdlib("c") }}
        - {{ compiler("fortran") }}
      host:
        - foo-devel

there's no way to make the "wrong" fortran compiler conflict with foo-devel, because we explicitly do not want a strong run-export from the general-purpose {{ compiler("fortran") }} on the compiler ABI. The only solution there would be that we add a host-export on _fortran_modules_abi to {{ compiler("fortran") }}; that would impose the right constraints in host, while avoiding too-tight constraints at runtime. The situation is explained/discussed in more detail in https://github.com/conda-forge/conda-forge.github.io/issues/2525.

W.r.t. naming, @isuruf said there

I would not call it host-exports, just another variant of run-exports as run-exports: strong also adds to host in addition to run.

I think I understand the thought behind that - it's certainly the easiest option to just add a third option to the existing run_exports: functionality, e.g.

build:
  run_exports:
    weak:       # equivalent to bare `run_exports: foo`
      - foo
    strong:
      - bar
    host_only:  # just for exposition, name can be bikeshed
      - baz

Personally, I think the name "run-export" is not ideal for something that only involves build & host environments. I also don't see the reason to have a strong/weak separation for host-exports (because a weak host-to-host export should just be a run-dependency of that package; this is the same argument made in the now-closed https://github.com/conda/conda-build/issues/3796 that's referenced in the OP).

Ultimately, the issue here is requirement-injection across environments. Usually build/host/run are independent of each other, but where they are not, we need to ensure that this is possible.

If we impose a single direction of influence from build -> host -> run, we have four possibilities

To 👉 From 👇	build:	host:	run:	Use-case	Current syntax
host:	-	-	✅	Shared libraries	run_exports: [foo] run_exports.weak: [foo]
build:	-	✅	✅	Constraint propagation	run_exports.strong: [foo]
build:	-	✅	-	Toolchain constraint	n/a
build:	-	-	✅	Compiler-runtimes	n/a

We currently have no way to express the third (topic of this issue), nor the 4th, though usually the 4th can be dealt with through a strong run-export.

The final consideration is that we're attaching metadata to a given output, without knowing in advance in which environment they will end up. In other words, what happens if a compiler (or something else with a strong run-export) ends up in a host: environment. Should it still export something to the run-environment?

For the "strong run-export", the case can be made that this should be the case (it's still a run-export, so it should end up in the run requirements, no matter if it starts out in host or build), but a package with a host-export that ends up in host: itself should not inject something into the run-requirements. For example, if a package (say foo) with a host-export (say bar) ends up in a host: environment of a third package (qux):

package: qux

requirements:
  build:
    - [...]
  host:
    - foo  # foo host-exporting bar should have no effect on qux
  run:
    - [...]

Here, either bar should be a runtime dependency of foo (if only foo, but not qux needs it), or qux should be explicit about its host-dependencies, i.e. add bar as a host-dependency.

This to me is the main argument against something like run_export.host_only, because the mechanics and semantics would be very confusing and unintuitive IMO.

---

As an alternative to bolting something on the side of run_exports and living with some inconsistencies there long-term, I think it would make sense to design this feature more coherently (at least for v1 recipes, which already structure run-exports differently -- under requirements: instead of under build:, though still following the same semantics otherwise).

In particular, I'm thinking about the following kind of structure

requirements:
  build:
    - [...]
  host:
    - [...]
  run:
    - [...]
  # relying on the surrounding "requirements" key for context
  export:
    host_to_run:            # matches weak run-export
      - a_shared_library
    build_to_host:          # "host-export"
      - a_build_constraint =*=*foo
    build_to_run:           # produces same effect as strong run-export when used together with build_to_host
      - a_compiler_runtime

I ran with the idea that v1 recipes are using already, of grouping all requirement-related things under requirements:, and I think, requirements.export is pretty self-explanatory. Rather than further overload the run_export terminology (which I believe preceded cb3 and then needed the strong/weak distinction to make up for it), I think it would be a good idea to be explicit about from where to where a requirement-export is expected to be applied.

Even though this makes the keys a bit longer (e.g. host_to_run:), this has the big advantage that it becomes unambiguous whether a given export should be applied, depending on which environment the package with the export finds itself in. For example, adding a compiler with a build_to_run export in some host: environment would not trigger any exports.

If we assume that host_to_run: is the 98% case, we could also allow an alias directly under export:, just as run_exports: foo and run_exports.weak: foo are equivalent today. In other words:

requirements:
  export:
    - libfoo        # possible extension: bare alias of host_to_run
# vs.
requirements:
  export:
    host_to_run:
      - libfoo      # equivalent to the above

Aside from the structure of the yaml keys, this is not as big a change as it may seem IMO; for example, the design would naturally maintain the semantics that requirements.export.*_to_run could still be collectively called “run exports” and requirements.export.build_to_host could informally be called “host export” without ambiguity.

The ignore side would only need minor modifications, i.e. removing _run from the current v1 formulation:

  requirements:
    ignore_exports:  # renamed from ignore_run_exports
      from_package:
        - zlib
      by_name:
        - libzlib

One thing this doesn't cover is exporting constraints rather than requirements, but that's also not handled by run-exports currently. If that's a thing that people want, we could discuss how to incorporate that into the design.

[wolfv]

Yeah I hate the "strong/weak" terminology as it hides what the run export actually does. And nobody really knows whether strong applies to host as well, or not. So I am very in favor of cleaning this up and come up with a flexible way of choosing the exports on a from-to basis.

[isuruf]

Overall I like the idea. I proposed the same thing when v1 was being developed, but there was no interest. I'm having a hard time finding the conversation though.

More things I would like the CEP to add are

host_to_constrains - to support `run_exports: weak_constrains`
build_to_constrains - to support `run_exports: strong_constrains`
build_to_build - to support transitive exports
host_to_host - to support transitive exports

[h-vetinari]

I just realized that I forgot a very important angle here. I was coming just from the recipe side, but we need to persist those different requirement-exports in the package metadata, which would also require modification of the repodata schema (c.f. e.g. #108). That makes it less feasible to do this only for v1 recipes, in the sense that conda{,-build} would have to at least be able to digest a new format of repodata.

[msarahan]

I like how clear this is. I'd be in favor of the CEP, and @isuruf's additions also make sense to me. The export to constraints especially seems useful. I'm not clear on build_to_build and host_to_host - why not just add these as dependencies?

nobody really knows whether strong applies to host as well, or not

It does! Or maybe not - you have me second guessing myself.

The run_exports name was something that definitely caused a lot of confusion. I think at the time, Ray and I only had the first two use cases in mind, with the end result applying to the run dependencies of the end package.

[isuruf]

I'm not clear on build_to_build and host_to_host - why not just add these as dependencies?

Because the constraints might not be strict. For eg: build 1 of B=1.* might depend on A=1.* and build 2 of B=1.* might depend on A=2.*, but any downstream of B (call it C)needs to link to both B=1.* and the version of A that it was compiled against.

This is useful when the headers of B uses headers of A and A transitively becomes a direct dependency of C even though it may not be obvious.

[AntoinePrv]

I second @isuruf comments on transitive host to host. This happens a lot in modern C++ libraries where the adoption of templates and contexpr functions push dependencies into public headers. It's currently pretty hard to understand (let alone explain) how they should be handled in the current situation.

[h-vetinari]

First draft (without specification, but including everything Isuru suggested) in #129