rattler-build icon indicating copy to clipboard operation
rattler-build copied to clipboard
verified publisher icon prefix-dev

race condition with `pin_subpackage`

Open pavelzw opened this issue 1 year ago • 2 comments 

# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json

context:
  name: my-package

recipe:
  name: ${{ name }}
  version: '1.0.0'

outputs:
  - package:
      name: ${{ name }}-${{ variant }}

  - package:
      name: ${{ name }}
    requirements:
      run:
        - ${{ pin_subpackage('my-package-a', exact=true) if variant == 'a' else pin_subpackage('my-package-b', exact=true) }}

variant:
  - a
  - b

Sometimes, it works:

❯ rattler-build build -r recipe/recipe.yaml -m recipe/variants.yaml

 ╭─ Finding outputs from recipe
 │ Found 4 variants
 │ Build variant: my-package--1.0.0-hcdbb9e0_0
 │ 
 │ ╭─────────────────┬───────────╮
 │ │ Variant         ┆ Version   │
 │ ╞═════════════════╪═══════════╡
 │ │ target_platform ┆ osx-arm64 │
 │ │ variant         ┆ a         │
 │ ╰─────────────────┴───────────╯
 │ Build variant: my-package-1.0.0-hbb13421_0
 │ 
 │ ╭─────────────────┬──────────────────╮
 │ │ Variant         ┆ Version          │
 │ ╞═════════════════╪══════════════════╡
 │ │ my-package-a    ┆ 1.0.0 hcdbb9e0_0 │
 │ │ target_platform ┆ osx-arm64        │
 │ │ variant         ┆ a                │
 │ ╰─────────────────┴──────────────────╯
 │ Build variant: my-package--1.0.0-h89e5c08_0
 │ 
 │ ╭─────────────────┬───────────╮
 │ │ Variant         ┆ Version   │
 │ ╞═════════════════╪═══════════╡
 │ │ target_platform ┆ osx-arm64 │
 │ │ variant         ┆ b         │
 │ ╰─────────────────┴───────────╯
 │ Build variant: my-package-1.0.0-h41b9c1b_0
 │ 
 │ ╭─────────────────┬──────────────────╮
 │ │ Variant         ┆ Version          │
 │ ╞═════════════════╪══════════════════╡
 │ │ my-package-b    ┆ 1.0.0 h89e5c08_0 │
 │ │ target_platform ┆ osx-arm64        │
 │ │ variant         ┆ b                │
 │ ╰─────────────────┴──────────────────╯
 │
 ╰─────────────────── (took 0 seconds)

 ╭─ Checking existing builds
 │
 ╰─────────────────── (took 0 seconds)

 ╭─ Running build for recipe: my-package-a-1.0.0-hcdbb9e0_0
 │
 │ ╭─ Fetching source code
 │ │ No sources to fetch
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running cache build
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Resolving environments
 │ │ 
 │ │ Finalized run dependencies: this output has no run dependencies
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running build script
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Packaging new files
 │ │ Copying done!
 │ │ Post-processing done!
 │ │ Writing test files
 │ │ Writing metadata for package
 │ │ Copying license files
 │ │ Copying recipe files
 │ │ Creating entry points
 │ │ 
 │ │ Files in package:
 │ │   - info/about.json
 │ │   - info/hash_input.json
 │ │   - info/index.json
 │ │   - info/paths.json
 │ │   - info/recipe/recipe.yaml
 │ │   - info/recipe/rendered_recipe.yaml
 │ │   - info/recipe/variant_config.yaml
 │ │   - info/recipe/variants.yaml
 │ │   - info/tests/tests.yaml
 │ │ Creating target folder "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64"
 │ │ Compressing archive...
 │ │ Archive written to "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-a-1.0.0-hcdbb9e0_0.conda"
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running package tests
 │ │ Removing previously cached package "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-a-1.0.0-hcdbb9e0_0"
 │ │ Creating test environment in "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/bld/rattler-build_my-package-a_1721894206/work/test"
 │ │ Collecting tests from "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-a-1.0.0-hcdbb9e0_0"
 │ │ ✔ all tests passed!
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 ╰─────────────────── (took 0 seconds)

 ╭─ Running build for recipe: my-package-1.0.0-hbb13421_0
 │
 │ ╭─ Fetching source code
 │ │ No sources to fetch
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running cache build
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Resolving environments
 │ │ 
 │ │ Finalized run dependencies:
 │ │ ╭──────────────────┬─────────────────────────╮
 │ │ │ Name             ┆ Spec                    │
 │ │ ╞══════════════════╪═════════════════════════╡
 │ │ │ Run dependencies ┆                         │
 │ │ │ my-package-a     ┆ ==1.0.0 hcdbb9e0_0 (PS) │
 │ │ ╰──────────────────┴─────────────────────────╯
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running build script
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Packaging new files
 │ │ Copying done!
 │ │ ⚠ warning Overdepending against my-package-a
 │ │ Post-processing done!
 │ │ Writing test files
 │ │ Writing metadata for package
 │ │ Copying license files
 │ │ Copying recipe files
 │ │ Creating entry points
 │ │ 
 │ │ Files in package:
 │ │   - info/about.json
 │ │   - info/hash_input.json
 │ │   - info/index.json
 │ │   - info/paths.json
 │ │   - info/recipe/recipe.yaml
 │ │   - info/recipe/rendered_recipe.yaml
 │ │   - info/recipe/variant_config.yaml
 │ │   - info/recipe/variants.yaml
 │ │   - info/tests/tests.yaml
 │ │ Creating target folder "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64"
 │ │ Compressing archive...
 │ │ Archive written to "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-1.0.0-hbb13421_0.conda"
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running package tests
 │ │ Removing previously cached package "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-1.0.0-hbb13421_0"
 │ │ Creating test environment in "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/bld/rattler-build_my-package_1721894206/work/test"
 │ │ Collecting tests from "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-1.0.0-hbb13421_0"
 │ │ ✔ all tests passed!
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 ╰─────────────────── (took 0 seconds)

 ╭─ Running build for recipe: my-package-b-1.0.0-h89e5c08_0
 │
 │ ╭─ Fetching source code
 │ │ No sources to fetch
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running cache build
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Resolving environments
 │ │ 
 │ │ Finalized run dependencies: this output has no run dependencies
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running build script
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Packaging new files
 │ │ Copying done!
 │ │ Post-processing done!
 │ │ Writing test files
 │ │ Writing metadata for package
 │ │ Copying license files
 │ │ Copying recipe files
 │ │ Creating entry points
 │ │ 
 │ │ Files in package:
 │ │   - info/about.json
 │ │   - info/hash_input.json
 │ │   - info/index.json
 │ │   - info/paths.json
 │ │   - info/recipe/recipe.yaml
 │ │   - info/recipe/rendered_recipe.yaml
 │ │   - info/recipe/variant_config.yaml
 │ │   - info/recipe/variants.yaml
 │ │   - info/tests/tests.yaml
 │ │ Creating target folder "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64"
 │ │ Compressing archive...
 │ │ Archive written to "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-b-1.0.0-h89e5c08_0.conda"
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running package tests
 │ │ Removing previously cached package "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-b-1.0.0-h89e5c08_0"
 │ │ Creating test environment in "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/bld/rattler-build_my-package-b_1721894206/work/test"
 │ │ Collecting tests from "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-b-1.0.0-h89e5c08_0"
 │ │ ✔ all tests passed!
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 ╰─────────────────── (took 0 seconds)

 ╭─ Running build for recipe: my-package-1.0.0-h41b9c1b_0
 │
 │ ╭─ Fetching source code
 │ │ No sources to fetch
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running cache build
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Resolving environments
 │ │ 
 │ │ Finalized run dependencies:
 │ │ ╭──────────────────┬─────────────────────────╮
 │ │ │ Name             ┆ Spec                    │
 │ │ ╞══════════════════╪═════════════════════════╡
 │ │ │ Run dependencies ┆                         │
 │ │ │ my-package-b     ┆ ==1.0.0 h89e5c08_0 (PS) │
 │ │ ╰──────────────────┴─────────────────────────╯
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running build script
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Packaging new files
 │ │ Copying done!
 │ │ ⚠ warning Overdepending against my-package-b
 │ │ Post-processing done!
 │ │ Writing test files
 │ │ Writing metadata for package
 │ │ Copying license files
 │ │ Copying recipe files
 │ │ Creating entry points
 │ │ 
 │ │ Files in package:
 │ │   - info/about.json
 │ │   - info/hash_input.json
 │ │   - info/index.json
 │ │   - info/paths.json
 │ │   - info/recipe/recipe.yaml
 │ │   - info/recipe/rendered_recipe.yaml
 │ │   - info/recipe/variant_config.yaml
 │ │   - info/recipe/variants.yaml
 │ │   - info/tests/tests.yaml
 │ │ Creating target folder "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64"
 │ │ Compressing archive...
 │ │ Archive written to "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-1.0.0-h41b9c1b_0.conda"
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running package tests
 │ │ Removing previously cached package "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-1.0.0-h41b9c1b_0"
 │ │ Creating test environment in "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/bld/rattler-build_my-package_1721894206/work/test"
 │ │ Collecting tests from "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-1.0.0-h41b9c1b_0"
 │ │ ✔ all tests passed!
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 ╰─────────────────── (took 0 seconds)

 ╭─ Build summary
 │
 │ ╭─ Build summary for recipe: my-package-a-1.0.0-hcdbb9e0_0
 │ │ Artifact: /private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-a-1.0.0-hcdbb9e0_0.conda (1.60 KiB)
 │ │ Variant configuration (hash: hcdbb9e0_0):
 │ │ ╭─────────────────┬───────────╮
 │ │ │ target_platform ┆ osx-arm64 │
 │ │ │ variant         ┆ a         │
 │ │ ╰─────────────────┴───────────╯
 │ │ 
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Build summary for recipe: my-package-1.0.0-hbb13421_0
 │ │ Artifact: /private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-1.0.0-hbb13421_0.conda (1.67 KiB)
 │ │ Variant configuration (hash: hbb13421_0):
 │ │ ╭─────────────────┬──────────────────╮
 │ │ │ my-package-a    ┆ 1.0.0 hcdbb9e0_0 │
 │ │ │ target_platform ┆ osx-arm64        │
 │ │ │ variant         ┆ a                │
 │ │ ╰─────────────────┴──────────────────╯
 │ │ 
 │ │ Run dependencies:
 │ │ ╭──────────────────┬─────────────────────────╮
 │ │ │ Name             ┆ Spec                    │
 │ │ ╞══════════════════╪═════════════════════════╡
 │ │ │ Run dependencies ┆                         │
 │ │ │ my-package-a     ┆ ==1.0.0 hcdbb9e0_0 (PS) │
 │ │ ╰──────────────────┴─────────────────────────╯
 │ │ 
 │ │ ⚠ warning Warnings:
 │ │ ⚠ warning Overdepending against my-package-a
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Build summary for recipe: my-package-b-1.0.0-h89e5c08_0
 │ │ Artifact: /private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-b-1.0.0-h89e5c08_0.conda (1.63 KiB)
 │ │ Variant configuration (hash: h89e5c08_0):
 │ │ ╭─────────────────┬───────────╮
 │ │ │ target_platform ┆ osx-arm64 │
 │ │ │ variant         ┆ b         │
 │ │ ╰─────────────────┴───────────╯
 │ │ 
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Build summary for recipe: my-package-1.0.0-h41b9c1b_0
 │ │ Artifact: /private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-1.0.0-h41b9c1b_0.conda (1.69 KiB)
 │ │ Variant configuration (hash: h41b9c1b_0):
 │ │ ╭─────────────────┬──────────────────╮
 │ │ │ my-package-b    ┆ 1.0.0 h89e5c08_0 │
 │ │ │ target_platform ┆ osx-arm64        │
 │ │ │ variant         ┆ b                │
 │ │ ╰─────────────────┴──────────────────╯
 │ │ 
 │ │ Run dependencies:
 │ │ ╭──────────────────┬─────────────────────────╮
 │ │ │ Name             ┆ Spec                    │
 │ │ ╞══════════════════╪═════════════════════════╡
 │ │ │ Run dependencies ┆                         │
 │ │ │ my-package-b     ┆ ==1.0.0 h89e5c08_0 (PS) │
 │ │ ╰──────────────────┴─────────────────────────╯
 │ │ 
 │ │ ⚠ warning Warnings:
 │ │ ⚠ warning Overdepending against my-package-b
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 ╰─────────────────── (took 0 seconds)

Sometimes it doesn't:

❯ rattler-build build -r recipe/recipe.yaml -m recipe/variants.yaml

 ╭─ Finding outputs from recipe
 │
 ╰─────────────────── (took 0 seconds)
Error:   × Missing output: my-package-a (used in pin_subpackage)

pavelzw avatar Jul 25 '24 07:07 pavelzw

Hm, interesting! Thanks for the reproducer, too! Unsure why this fails.

wolfv avatar Aug 01 '24 12:08 wolfv

I think this one is related to #985 - because we migth not get the "right" name when ordering the packages this fails randomly. So I do think that we should retry this one when we fix #985.

wolfv avatar Aug 01 '24 13:08 wolfv

found a smaller reproducer

# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json

recipe:
  name: my-package
  version: '1.0.0'

outputs:

  - package:
      name: my-package
    requirements:
      run:
        - ${{ pin_subpackage('my-package-a') }}

  - package:
      name: my-package-${{ variant }}

variant:
  - a

pavelzw avatar Oct 21 '24 14:10 pavelzw

Another reproducer (not really smaller as the last one):

# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json

context:
  name: package
  version: '0.1.0'

recipe:
  name: ${{ name }}-split
  version: ${{ version }}

outputs:
  - package:
      name: package
    requirements:
      run:
        - if: add_subpackage == 'true'
          then:
            - ${{ pin_subpackage('package-2', exact=true) }}
  - package:
      name: package-2

add_subpackage:
  - true

$ rattler-build build -r conda.recipe/recipe.yaml -c conda-forge -m conda.recipe/conda_build_config.yml
Error:   × Missing output: package-2 (used in pin_subpackage)

pavelzw avatar Nov 28 '24 14:11 pavelzw

should be fixed with the new variant rendering

pavelzw avatar Dec 23 '24 11:12 pavelzw