Specifying a toolchain more than once in the same root spec doesn't work correctly if it contains unconditional dependencies

[jgraciahlrs]

Steps to reproduce

Trying to concretize an environment I got the error mentioned above.

Error message and files mentioned therein are attached below.

Error message

eval $(spack -C $SPACK_ROOT/../config-hlrs env activate hlrs-error --sh)
spack --debug --stacktrace -C $SPACK_ROOT/../config-hlrs concretize --force --fresh &> $SPACK_ENV/concretize.log

The file concretize.log is attached.

Files mentioned in the error message: input-1.json input-2.json input-3.json input-4.json input-5.json input-6.json output-1.json output-2.json output-3.json output-4.json output-5.json output-6.json

Information on your system

spack debug report
* **Spack:** 1.1.0 (https://github.com/spack/spack/commit/0c2be44e4ece21eb091ad5de4c97716b7c6d4c87)
* **Builtin repo:** not available
* **Python:** 3.11.7
* **Platform:** linux-rhel9-zen4

General information

• [x] I have run spack debug report and reported the version of Spack/Python/Platform
• [x] I have searched the issues of this repo and believe this is not a duplicate
• [x] I have run the failing commands in debug mode and reported the output

[alalazo]

It would be good if you could give us a reproducer environment. I see in the logs that the solver concretized to a cray-mpich %c=rocmcc when the request seemed to be cray-mpich %c=cce but it's hard to diagnose further without seeing the failing concretization.

Do you have a spack.yaml to share, that could exhibit this issue?

[jgraciahlrs]

Thanks for looking into this. I do no longer have the original case, mainly because our Spack config has changed since then.

However, the following case shows the same behaviour.

• 20251212-spack.yaml with additions from toolchains.yaml and packages.yaml in comments
• 20251210-concretize.log output of concretize with stacktrace
• 20251210-input-1.json
• 20251210-output-1.json

[alalazo]

Thanks for the reproducer. There is indeed a bug in Spack, because your configuration is unsat - so the concretizer shouldn't error like this, but point you to the fact that you are trying to attach %fortran unconditionally to cray-mpich.

This is because in your toolchain definition you have:

toolchains:
    cray:
    - spec: "%c=cce"
      when: "%c"
    - spec: "%cxx=cce"
      when: "%cxx"
    - spec: "%fortran=cce"

the last line should instead be:

    - spec: "%fortran=cce"
      when: "%fortran"

Also, in you externals it's better to use the actual compiler, not the toolchain:

- spec: [email protected]+wrappers %cce # not %cray
  prefix: /opt/cray/pe/mpich/8.1.33/ofi/crayclang/20.0

With these two changes I get the following spec:

which seems consistent with your request to use gcc@11 where possible, apart from the mpip and cray-mpich nodes.

I'll follow on the wrong error and close the report once I have a solution for that.

[alalazo]

Here's a minimal reproducer for the error:

spack.yaml

spack:
  toolchains:
    cray:
    - spec: "%c=cce"
      when: "%c"
    - spec: "%cxx=cce"
      when: "%cxx"
    - spec: "%fortran=cce"
    core:
    - spec: "%[email protected]"
      when: "%c"
    - spec: "%[email protected]"
      when: "%cxx"
    - spec: "%[email protected]"
      when: "%fortran"
  packages:
    all:
      prefer:
      - '%core'
    cce:
      externals:
      - spec: [email protected]
        prefix: /opt/cray/pe/cce/20.0.0
        extra_attributes:
          compilers:
            c: /usr/bin/gcc
            cxx: /usr/bin/g++
            fortran: /usr/bin/gfortran
    gcc:
      externals:
      - spec: [email protected] languages:='c,c++,fortran'
        prefix: /usr
        extra_attributes:
          compilers:
            c: /usr/bin/gcc
            cxx: /usr/bin/g++
            fortran: /usr/bin/gfortran
    cray-mpich:
      externals:
      - spec: [email protected]+wrappers %cce
        prefix: /opt/cray/pe/mpich/8.1.33/ofi/crayclang/20.0
  specs:
  - mpip %cray ^ cray-mpich %cray
  view: false
  concretizer:
    unify: when_possible
    reuse: false

The issue doesn't happen if the toolchain gets expanded in the configuration. It seems it's due to this check:

https://github.com/spack/spack/blob/8a47f8d8ea1a3a0d41ff8acf5e74a9ec3cb4e8c5/lib/spack/spack/solver/asp.py#L2523

where in the wrong case (the one using %cray everywhere) the cce that gets expanded for cray-mpich has the same id as the one used for mpip - and thus a few facts are skipped.

[alalazo]

For the record I'm also attaching the facts I got by expanding manually the toolchain vs. using the reproducer:

lp_files.tar.gz

Here one can see that a few facts are missing in one case

[alalazo]

Hotfix for the bug:

diff --git a/lib/spack/spack/spec_parser.py b/lib/spack/spack/spec_parser.py
index bfdabe17f0..8c093ce01b 100644
--- a/lib/spack/spack/spec_parser.py
+++ b/lib/spack/spack/spec_parser.py
@@ -461,10 +461,13 @@ def _apply_toolchain(
             toolchain = self._parse_toolchain(name)
             self.parsed_toolchains[name] = toolchain
 
-        toolchain = self.parsed_toolchains[name]
-        if propagation == PropagationPolicy.PREFERENCE:
-            toolchain = toolchain.copy(propagation=propagation)
+        if propagation != PropagationPolicy.PREFERENCE:
+            propagation = None
 
+        # Here we need to copy because we want "foo %toolc ^bar %toolc" to generate different
+        # objects for the toolc attached to foo and bar, since the solver depends on that to
+        # generate facts
+        toolchain = self.parsed_toolchains[name].copy(propagation=propagation)
         spec.constrain(toolchain)
 
     def _parse_toolchain(self, name: str) -> "spack.spec.Spec":

[alalazo]

Can you check #51731? With that you should obtain a more sensible error message, similar to:

$ spack -e . concretize 
==> Error: failed to concretize `mpip %[when='%c'] c=cce %[when='%cxx'] cxx=cce %fortran=cce ^cray-mpich %[when='%c'] c=cce %[when='%cxx'] cxx=cce %fortran=cce` for the following reasons:
     1. Cannot build cray-pmi, since it is configured `buildable:false` and no externals satisfy the request
     2. Cannot build cray-mpich, since it is configured `buildable:false` and no externals satisfy the request
     3. cray-mpich cannot have a dependency on fortran

[jgraciahlrs]

Interesting. The missing conditional in

    - spec: "%fortran=cce"

is just a copy & paste error. It is actually present in my config.

    - spec: "%fortran=cce"
      when: "%fortran"

Nonetheless, I can confirm that #51731 results in a nicer error message (for the unconditional case.

Moreover, #51731 also fixes my actual problem where the spec contains:

 specs:
  - matrix:
    - [mpip]
    - ['%cray ^cray-mpich%cray', '%amd ^cray-mpich%amd'] # force matching compilers for mpi

Cheers!