Spec discussion

[wolfv]

Notes from an virtual spec discussion we just had over Zoom:

Other syntax for sel(...)

Replace

- sel(linux): xtensor

with

- if: linux
   put: xtensor

(this should also work for list, e.g.

- if: linux
  put:
  	- xtensor
  	- xsimd
  	- ...

To skip certain steps, we will use

    if:
      - unix and py > "3.6"
      - unix and py > ver("3.6")
      - unix and vercmp(py, "3.6")
    # implies `put: {skip: true}` when used at a top-level step key

The recursive parsing logic for this will stay exactly the same.

We can extend that eventually to for loops:

- for: element in mylist
   put: {{ element[0:10] }}

Rename outputs to tasks or steps

And furthermore, make sure that _not each step has to produce a package!

A package is only produced if the step has a package key. Otherwise, there can be build output produced (for later splitting) or it can be used to run non-output specific tests (the tests can require other outputs from the recipe).

steps:
- step:
    name: buildstep
  requirements:
    build:
      - "{{ compilers('cxx') }}"
    host:
      - libcpp
      - libxnd
- package:
    name: libx
    version: "1.2.3"
  requirements:
    stepresult:
      # note: this needs to make sure that the 
      #.      proper variant is selected!
      - buildstep
    run:
      - something
    test:
      - otherpkg
  test:
    commands:
      - checklibx
- test:
    name: supertest
    commands:
      - pip check
  requirements:
    test:
      - libx

Replace Jinja functions with YAML dictionary

Instead of writing {{ compiler('cxx') }} it was suggested to write compiler: cxx.

Other currently used functions:

# All currently availble jinja functions in boa:

- compiler: cxx
- compiler: fortran
- cdt: xorg-libx11
- run_exports:
    - pin_subpackage:
        name: "{{ name }}"
        # default optional values below
        max_pin: "x.x.x.x.x"
        exact: false
requirements:
  run:
    - pin_compatible:
        name: xtensor
        # default values below
        lower_bound: null
        upper_bound: null
        min_pin: "x.x.x.x.x.x"
        max_pin: "x"
        exact: false

# this is pretty actually useless, we should probably keep it a jinja function or put all env vars into the context to make them usable from jinja anyways
# although we could have a special section in `context` where you can add env vars (similar to github actions)
- environ: MYXPKG_VERSION

[wolfv]

I went ahead and test-implemented the steps syntax to something like this:

Recipe with multiple steps + splitting

context:
  name: multistep
  version: 0.7.3

package:
  name: '{{ name|lower }}'
  version: '{{ version }}'

build:
  number: 0

steps:
  - step:
      name: buildstep
    build:
      script:
        - echo "File 1" > file_1.txt
        - echo "File 2" > file_2.txt
        - echo "File 3" > file_3.txt

  - package:
      name: multistep-a
      version: 1.2.3
    source:
      - step: buildstep
    build:
      script:
        - cp file_1.txt $PREFIX/file_1.txt

  - package:
      name: multistep-b
      version: 1.2.3
    source:
      - step: buildstep
    build:
      script:
        - cp file_2.txt $PREFIX/file_2.txt
        - cp file_3.txt $PREFIX/file_3.txt

However, I also noticed some issues, especially around variants, requirements & run_exports. I am working on additional visualizations right now so we can make sure that we get it right. Basically, the dependee needs to inherit variants & run exports and stuff like this.

[wolfv]

I think it would be a fine idea to explicitly add environment variables into the recipe -- maybe as part of the context. An additional benefit is that we can use them to compute variants and replace the hacky regex-approach taken in conda-build to find "used variants" with something explicit. I need to find the right place(s) to do this. There is also the script_env key which we could use for this purpose.

[isuruf]

Everything looks good to me. The keyword put doesn't feel right, but I couldn't think of a better name other than then.

[isuruf]

To skip certain steps, we will use

I don't understand this part.

(this should also work for list, e.g.

- if: linux
  put:
  	- xtensor
  	- xsimd
  	- ...

How would this work if we need a list of lists?

[wolfv]

@mariusvniekerk just pointed out that the documentation currently cuts off in a slighly awkward place when talking about defined "env" variables. I think boa is currenty missing some Jinja vars like:

• PYTHON
• R
• ...

we should make a list and figure out which are absolutely needed and add them to the context.

[jaimergp]

I think all the env vars here are also accepted as jinja vars

[tdegeus]

I just encountered something that could be useful here (discussion on gitter with @jaimergp ). It would be nice if one can append global build settings in a multi-output feedstock. I.e., this works:

build:
  number: 0

outputs:

  - name: foo
    build:
      script_env: SETUPTOOLS_SCM_PRETEND_VERSION={{ version }}
    ...

  - name: python-foo
    build:
      script_env: SETUPTOOLS_SCM_PRETEND_VERSION={{ version }}
      skip: true  # [python_impl == 'pypy']
    ...

and this works (albeit a bit buggy w.r.t. to rendering, but that's not the point here):

build:
  number: 0
  script_env: SETUPTOOLS_SCM_PRETEND_VERSION={{ version }}
  skip: true  # [python_impl == 'pypy']

outputs:

  - name: foo
     ...

  - name: python-foo
    ...

but this does not work

build:
  number: 0
  script_env: SETUPTOOLS_SCM_PRETEND_VERSION={{ version }}

outputs:

  - name: foo
    ...

  - name: python-foo
    build:
      skip: true  # [python_impl == 'pypy']
    ...

whereas I think the latter is the most elegant.

[isuruf]

That's a backward incompatible change that'll be hard to debug if it was changed.