Multiple outputs - implement the cache output

[wolfv]

We have gotten rid of the buggy "top-level" output that conda-build implemented. However, it is quite useful to have a single build step and then split a package into multiple sub-pacakges to save compute resources (instead of starting from scratch for every output).

For this, our CEP proposal includes a special output type that would not produce a final package, but rather only a cache package that can be reused by later outputs. However, we haven't fully worked out how it would act.

I am listing some of the open questions:

• Do we restore both everything that the cache step installed into the $PREFIX as well as what is in the current "work dir"? When we restore the work dir, a second cmake run would be fast
• Do we copy any requirements / run dependencies (through run exports) / ... from the cache build?
• Do we implement the files: [glob] way of splitting a package? That would mean the cache step puts a bunch of things in $PREFIX, and subsequent steps take only parts of it

Would love to hear the communities opinions on these items. The current proposal for the cache-only packages looks something like this:

outputs:
  # this first package is "cache-only"
  - package: 
      name: test-split
    build:
      cache-only: true
  # the next two packages start exactly where `test-split` left off
  - package:
      name: libtest
    build:
      needs-cache: [test-split]
  - package:
      name: test
    build:
      needs-cache: [test-split]

[wolfv]

There is a little more thinking going on in the CEP: https://github.com/conda-incubator/ceps/blob/18ce1875c4758347d83e913012ec48fb59e381df/cep-20.2.md#outputs-section

[wolfv]

I think (my) current idea is slightly different.

We would only have a single top-level "output" that is under a cache key:

cache:
  build: ...
  requirements: ...

outputs:
  - package: ... 
 

[wolfv]

I am working on this example recipe:

recipe:
  name: cachycache
  version: 1.0.0

cache:
  build:
    script:
      - mkdir -p $PREFIX/include
      - mkdir -p $PREFIX/lib
      - touch $PREFIX/include/include_file.h
      - touch $PREFIX/include/include_file.c
      - touch $PREFIX/include/include_file.exe
      - touch $PREFIX/lib/lib_foo.so

    include_files:
      - include/
      - lib/

  requirements:
    build:
      - python
    host:
      - python

outputs:
  - package:
      name: cachycache-headers
    build:
      files:
        - include/*

  - package:
      name: cachycache-lib
    build:
      files:
        - lib/*

For a simple "split"-package.

[wolfv]

The initial implementation is merged but it still contains some bugs (notably with prefix replacement) and is only available under the --experimental flag: #898

[wolfv]

We can close this for the time being and continue finishing up the cache implementation.