conda-forge: CUDA 12 / new CUDA SDK bringup

[jakirkham]

Topic

Opening this issue as a central point for information, discussion, etc. around the CUDA 12 migration and the new style CUDA SDK packages.

What follows is background about how CUDA builds are supported now, what is changing about them (and why), and what users can do to adopt the new best practices.

Background

As a little bit of background, the conda-forge CUDA builds up to this point have relied on a few key components:

• nvidia/cuda (here's one template image as an example)
• The CUDA redistributable libraries combined together in the cudatoolkit package (for linkage into other packages built in conda-forge)
• The nvcc wrapper script metapackage to glue the previous two pieces together during a build

There are of course plenty more pieces like how these components are used in the global pinnings, how CUDA migrators work, how other infra around CUDA works, how tooling configures CUDA builds (in staged-recipes and feedstocks).

However the big picture of Docker images, CUDA libraries, and a wrapper script to bind them is the gist. This has worked ok for a while, but there are different needs it didn't satisfy.

Feedback

Based on user feedback on this model over the years, we have learned a few key things:

1. The cudatoolkit package is bulky ( for example https://github.com/conda-forge/arrow-cpp-feedstock/issues/962 )
2. There are other components users would like access to ( https://github.com/conda-forge/cudatoolkit-feedstock/issues/72 )
3. The current model has also created integration pains at multiple points

Follow-up

Based on this feedback, we started exploring a new model for handling these packages ( https://github.com/conda-forge/cudatoolkit-feedstock/issues/62 ). After a bit of iterating with various stakeholders, we felt ready to start packaging all of these components ( https://github.com/conda-forge/staged-recipes/issues/21382 ). This admittedly has been a fair bit of work and we appreciate everyone's patience while we have been working through it (as well as everyone who shared feedback). Though it looks like we are nearing the end of package addition and moving on to next steps.

Structure

Compiler

One particular component that has taken a fair amount of effort to structure correctly has been the compiler package. However after some final restructuring ( https://github.com/conda-forge/cuda-nvcc-feedstock/issues/12 ) and some promising test results on a few CUDA-enabled feedstocks, this seems to be working in the right way. There are a few particular points of note about how the compiler package is structured (and how correspondingly other CTK packages are structured).

The compiler package is designed to be used in cross-compilation builds. As a result all headers and libraries live in a targets-based structure. This is best observed in this code from cuda-nvcc:

[[ "@cross_target_platform@" == "linux-64" ]] && targetsDir="targets/x86_64-linux"
[[ "@cross_target_platform@" == "linux-ppc64le" ]] && targetsDir="targets/ppc64le-linux"
[[ "@cross_target_platform@" == "linux-aarch64" ]] && targetsDir="targets/sbsa-linux"

As a result $PREFIX/$targetsDir contains the needed headers, libraries, etc. for building for that target platform. The right paths should already be picked up by nvcc and gcc (the host compiler). So please let us know if that is not happening for some reason.

CUDA Toolkit Packages

As noted previously, cudatoolkit does not exist in this new model (starting with CUDA 12.0). Also no libraries are pulled in outside of cudart (in particular the static library). So other libraries like libcublas, libcurand, etc. need to explicitly be added to requirements/host for CUDA 12. In particular there are *-dev libraries (like libcublas-dev) that would need to be added to host. The -dev package contains both headers and shared libraries. Also the -dev packages add run_exports for their corresponding runtime library; so, that dependency will be satisfied automatically using the usually conda-build behavior.

Building off the compiler section above, the move to support cross-compilation means that packages supply headers, libraries, etc. under a different path than $PREFIX. Instead the paths look something like this (using $targetsDir from above):

• Headers: $PREFIX/$targetsDir/include
• Libraries: $PREFIX/$targetsDir/lib
• Stub libraries (if any): $PREFIX/$targetsDir/lib/stubs

Note that these paths shouldn't be needed anywhere since nvcc and gcc know about them (when {{ compiler("cuda") }} is added to requirements/build). Though if some issues come up related to this, please reach out.

CUDA version alignment

Previously the cudatoolkit package had served a CUDA version alignment function ( https://github.com/conda-forge/conda-forge.github.io/issues/687 ). Namely installing cudatoolkit=11.2 would ensure all packages supported CUDA 11.2. This could be useful when trying to satisfy some hardware constraints.

Another issue the cudatoolkit package solved, which may be less obvious at the outset, is it provides a consistent set of libraries that are all part of the same CUDA release (say 11.8 for example). This is trivially solved with one package. However splitting these out into multiple packages (and adding more packages as well), makes this problem more complicated. The cuda-version package also comes in here by aligning all CUDA packages to a particular CUDA version.

Finally as there is a fissure between the CUDA 11 & 12 worlds in terms of how these CUDA packages are handled, the cuda-version mends this gap. In particular cuda-version constrains cudatoolkit and cuda-version is backported to all CUDA version conda-forge has shipped. As a result cuda-version can be used in both CUDA 11 & 12 use cases to smooth over some of these differences and standardize CUDA version handling.

More details about the cuda-version package can be found in this README.

Final Observations

It's worth noting this new structure differs from notably from the current one in conda-forge, where we have Docker images that place the full CUDA Toolkit install in /usr/local/cuda and runtime libraries added automatically. This is done to both address use cases where only cudart or nvcc are needed (with no additional dependencies) as well as use cases where extra features are needed. Trying to address both of these means greater ability to fine-tune use cases, but it also means a bit more work for maintainers. Hoping this provides more context around why these changes are occurring and what they mean for maintainers and users.

Next Steps

Recently the CUDA 12 migrator was merged ( https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/4400 ). Already a handful of CUDA 12 migrator PRs have gone out.

For feedstocks that statically link cudart (default behavior) and don't use other libraries, there may be nothing additional to do.

For feedstocks that depend on other CUDA Toolkit components (say cuBLAS and cuRAND), those can be added to host with a selector like so...

diff --git a/recipe/meta.yaml b/recipe/meta.yaml
--- a/recipe/meta.yaml
+++ b/recipe/meta.yaml
@@ -2,2 +2,5 @@
  host:
     - python
     - pip
     - numpy
+    - libcublas-dev      # [(cuda_compiler_version or "").startswith("12")]
+    - libcurand-dev      # [(cuda_compiler_version or "").startswith("12")]

Similar patterns can be followed for other dependencies. Also please feel free to reach out for help here if needed

On CUDA 12.x

Initially we just added CUDA 12.0.0. We now are adding CUDA 12.x releases (with x > 0). In particular we have added CUDA 12.1 ( https://github.com/conda-forge/cuda-feedstock/issues/11 ) and are working on CUDA 12.2 ( https://github.com/conda-forge/cuda-feedstock/issues/13 )

If users want to pin tightly to a specific CUDA 12.x version during the build, would recommend adding cuda-version in the relevant requirements sections. For example to constrain during the build (though not necessarily at runtime)...

requirements:
  build:
    ...
    - {{ compiler('cuda') }}                    # [(cuda_compiler or "None") != "None"]
  host:
    ...
    - cuda-version {{ cuda_compiler_version }}  # [(cuda_compiler_version or "None") != "None"]
  run:
    ...

Thanks all! 🙏

[h-vetinari]

Thanks for all the work on this, this is amazing! :)

[mikemhenry]

Yes, thank you all! I am very excited to start updating packages to work with CUDA 12 and I really like how things are done now, much easier to work with, but now it will be quite a pain to support packages with the old CUDA 10 & 11 methods and the new CUDA 12 ones. Any idea when conda-forge will drop CUDA 10 & 11 support?

[jakirkham]

Thanks all! 🙏

We are really happy that these tools are now getting into your hands 😄

Any idea when conda-forge will drop CUDA 10 & 11 support?

This is a good question

Would be in favor of dropping CUDA 10 sooner (many feedstocks already opt-out anyways). Had started this previously ( https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/1708 ). Though think folks weren't ready to drop CUDA 10 yet. Maybe that has changed since. We can revisit

CUDA 11 support will be probably still be around for a while (at least until more libraries and infra) have moved to CUDA 12. Maybe we can think about which CUDA 11.x versions to build for (and maybe drop some older ones?)

Have tried to minimize pain like smoothing over CUDA 11 / 12 compiler differences under-the-hood ( https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/4407 ). We can try to simplify the conditional logic a bit ( https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/4534 ). Would welcome other suggestions on how we can make the CUDA 11 / 12 maintenance effort easier 🙂

[ngam]

Issue: Many packages use the environment variable CUDA_HOME to point to the root of where the tools used to be. Question: What's a good replacement for this now? I suppose CONDA_PREFIX or PREFIX, right? Or $PREFIX/targets/${target_platform}?

[ngam]

Another question:

2. There are other components users would like access to ( ptxas executable cudatoolkit-feedstock#72 )

Where is ptxas located now? We need it for tensorflow and jaxlib.

[ngam]

Finally, thanks a lot for this! 😸

I would recommend we curate a few examples in this issue to keep track of how people adapt to CUDA 12. I am working on the jaxlib rebuild (and soon after that tensorflow and maybe pytorch). I updated the compute targets already, but running into some issues... https://github.com/conda-forge/jaxlib-feedstock/pull/181; would be good if some experts can help out :)

[ngam]

Building off the compiler section above, the move to support cross-compilation means that packages supply headers, libraries, etc. under a different path than $PREFIX. Instead the paths look something like this (using $targetsDir from above):

• Headers: $PREFIX/$targetsDir/include
• Libraries: $PREFIX/$targetsDir/lib
• Stub libraries (if any): $PREFIX/$targetsDir/lib/stubs

Note that these paths shouldn't be needed anywhere since nvcc and gcc know about them (when {{ compiler("cuda") }} is added to requirements/build). Though if some issues come up related to this, please reach out.

Can you clarify this?

It seems to me that adding "when {{ compiler("cuda") }} is added to requirements/build", the result is that the headers are under $BUILD_PREFIX/$targetsDir/include (not $PREFIX). Is that what we want here or is that the intended change? Do we want these headers included under the build_env or the host_env?

[jakirkham]

Issue: Many packages use the environment variable CUDA_HOME to point to the root of where the tools used to be. Question: What's a good replacement for this now? I suppose CONDA_PREFIX or PREFIX, right? Or $PREFIX/targets/${target_platform}?

Would use $PREFIX. Though as a splayed layout is used, some things are split between $BUILD_PREFX & $PREFIX (as noted below). As a result some things like nvcc may need to be overridden specifically

2. There are other components users would like access to ( ptxas executable cudatoolkit-feedstock#72 )

Where is ptxas located now? We need it for tensorflow and jaxlib.

When adding {{ compiler("cuda") }} to requirements/build, it will be added to $BUILD_PREFIX/bin (along with other executables)

I would recommend we curate a few examples in this issue to keep track of how people adapt to CUDA 12.

Yeah migrator PRs are linking to this thread. So there may already be a few to look at. Here's a more comprehensive search

A goal at the end of this is developing docs ( https://github.com/conda-forge/conda-forge.github.io/issues/1927 ). Though that may be a function of what things wind up being generally relevant (and any tweaks made to tooling)

I updated the compute targets already, but running into some issues... https://github.com/conda-forge/jaxlib-feedstock/pull/181; would be good if some experts can help out :)

Noted. Will try to look.

It seems to me that adding "when {{ compiler("cuda") }} is added to requirements/build", the result is that the headers are under $BUILD_PREFIX/$targetsDir/include (not $PREFIX). Is that what we want here or is that the intended change? Do we want these headers included under the build_env or the host_env?

For now there are some compiler headers there. We plan to split these out at a future date ( https://github.com/conda-forge/cuda-nvcc-impl-feedstock/issues/4 ). The rest of the headers (from say libcublas-dev) are in $PREFIX

There are flags included to look in $BUILD_PREFIX/$targetsDir/include when relevant. So this shouldn't be a thing that recipes to need specify (if so that is an issue that would be good to mention here with an example and details)

[ngam]

Thanks! I think the issue with jaxlib and tensorflow is going to be simply resolving the paths (CUDA_HOME needs fixing). I will follow up later when I make more progress

[hadim]

Probably a very naive question but I am wondering how are we supposed to consume this new cuda 12? I don't see cudatoolkit for v12. Should we use the meta cuda-version package?

[hadim]

Hum it looks like I found the answer at https://github.com/conda-forge/cuda-version-feedstock/blob/main/recipe/README.md#for-users-of-gpu-packages, that seems to indicate that indeed cuda-version is the way to go.

[jakirkham]

Yep that's right. There is a note about this in the OP under "CUDA version alignment" (though recognize there is a lot to parse there)

Should add cuda-version has been backported to all CUDA versions that conda-forge has supported. So it is the best way forward (even when working with CUDA 11)

[ngam]

@jakirkham I am still unclear about one point: Are we supposed to use the new packaging format with only CUDA 12+ or can we use with CUDA 11.2?

I am asking because I see something this like in Jax which indicates they're using the new packaging format with 11.x, and I am trying to catch up in the jaxlib-feedstock:

        'cuda11_pip': [
          f"jaxlib=={_current_jaxlib_version}+cuda11.cudnn{_default_cuda11_cudnn_version}",
          "nvidia-cublas-cu11>=11.11",
          "nvidia-cuda-cupti-cu11>=11.8",
          "nvidia-cuda-nvcc-cu11>=11.8",
          "nvidia-cuda-runtime-cu11>=11.8",
          "nvidia-cudnn-cu11>=8.8",
          "nvidia-cufft-cu11>=10.9",
          "nvidia-cusolver-cu11>=11.4",
          "nvidia-cusparse-cu11>=11.7",
        ],


        'cuda12_pip': [
          f"jaxlib=={_current_jaxlib_version}+cuda12.cudnn{_default_cuda12_cudnn_version}",
          "nvidia-cublas-cu12",
          "nvidia-cuda-cupti-cu12",
          "nvidia-cuda-nvcc-cu12",
          "nvidia-cuda-runtime-cu12",
          "nvidia-cudnn-cu12>=8.9",
          "nvidia-cufft-cu12",
          "nvidia-cusolver-cu12",
          "nvidia-cusparse-cu12",
        ],

My plan in the coming weeks is to simply migrate to CUDA 12 and drop CUDA 11.x in feedstocks that I am working in (unless users object; I will give users and other maintainers enough time to decide before actually moving). So, if we cannot use the new packaging format with 11.x, this will probably accelerate my move towards CUDA 12 only going forward. It goes without saying, that I am super excited and happy about this effort (CUDA 12) that you and the team put together!

[jakirkham]

Are we supposed to use the new packaging format with only CUDA 12+ or can we use with CUDA 11.2?

Only CUDA 12+ is covered by the new package format in conda-forge

Recognize this causes some maintenance burden

Unfortunately there are some challenges backporting these packages to CUDA 11 beyond building all the feedstocks for CUDA 11 (which is a significant amount of work on its own), but also making sure it is done in a way that doesn't break existing packages & installs using the legacy format. So we've opted to make the breaking change occur with a CUDA major version bump to make it clearer that there are significant incompatible changes

I am asking because I see something this like in Jax which indicates they're using the new packaging format with 11.x

IIUC those are referring to wheels from a 3rd party index. Though Jax devs would likely know better what that means

My plan in the coming weeks is to simply migrate to CUDA 12 and drop CUDA 11.x in feedstocks that I am working in (unless users object; I will give users and other maintainers enough time to decide before actually moving).

One thing to be aware of is there are still a bunch of CSPs on CUDA 11. Not that you should feel obligated to support them. Just something to be aware of

Might be worth raising an issue on the feedstock to indicate this intent to drop CUDA 11 and move to CUDA 12

So, if we cannot use the new packaging format with 11.x, this will probably accelerate my move towards CUDA 12 only going forward.

Understood. If there are other pain points (beyond the package format difference), would be interested to know what they are

It goes without saying, that I am super excited and happy about this effort (CUDA 12) that you and the team put together!

Glad to hear that! Hoping this improves the CUDA packaging experience going forward 🙂

[jakirkham]

JFYI we have added CUDA 12 support for other architectures with PR: https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/4799

For any feedstocks that support other architectures (and was already migrated to CUDA 12), the bot will submit a new CUDA 12 migration PR for arch support

Any feedstocks that have a CUDA 12 migration PR open, but would like to add CUDA 12 for other architectures, please ask the bot to re-run. The bot will then close out the CUDA 12 migration PR and add a new one with the other architectures included

For feedstocks still waiting to be migrated, no action is needed

Please let us know if you have any questions 🙂

[carterbox]

I try to compile a simple test program in the test section of my recipe. CMake is having trouble finding the CUDAToolkit in the test environment when cross-compiling on linux.

+ cmake -GNinja -DCMAKE_AR=/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/bin/powerpc64le-conda-linux-gnu-ar -DCMAKE_CXX_COMPILER_AR=/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/bin/powerpc64le-conda-linux-gnu-gcc-ar -DCMAKE_C_COMPILER_AR=/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/bin/powerpc64le-conda-linux-gnu-gcc-ar -DCMAKE_RANLIB=/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/bin/powerpc64le-conda-linux-gnu-ranlib -DCMAKE_CXX_COMPILER_RANLIB=/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/bin/powerpc64le-conda-linux-gnu-gcc-ranlib -DCMAKE_C_COMPILER_RANLIB=/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/bin/powerpc64le-conda-linux-gnu-gcc-ranlib -DCMAKE_LINKER=/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/bin/powerpc64le-conda-linux-gnu-ld -DCMAKE_STRIP=/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/bin/powerpc64le-conda-linux-gnu-strip -DCMAKE_BUILD_TYPE=Release -DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER -DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY -DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY '-DCMAKE_FIND_ROOT_PATH=/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold;/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_build_env/powerpc64le-conda-linux-gnu/sysroot' -DCMAKE_INSTALL_PREFIX=/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold -DCMAKE_INSTALL_LIBDIR=lib '-DCMAKE_PROGRAM_PATH=/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_build_env/bin;/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/bin' '-DCMAKE_FIND_ROOT_PATH=/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold;/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_build_env/powerpc64le-conda-linux-gnu/sysroot;/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/targets/ppc64le-linux;/home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_build_env/targets/ppc64le-linux' .
-- Unable to find cuda_runtime.h in "$PREFIX/include" for CUDAToolkit_INCLUDE_DIR.
-- Unable to find cublas_v2.h in either "" or "/home/conda/feedstock_root/build_artifacts/math_libs/include"
CMake Error at /home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find CUDAToolkit (missing: CUDAToolkit_INCLUDE_DIR)
Call Stack (most recent call first):
  /home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /home/conda/feedstock_root/build_artifacts/kokkos_1693411108130/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/share/cmake-3.27/Modules/FindCUDAToolkit.cmake:938 (find_package_handle_standard_args)
  CMakeLists.txt:1 (find_package)

Should I expect this to work/not work out of the box? Or do I need to manually add the targetsDir paths as mentioned in this issue.

https://github.com/conda-forge/kokkos-feedstock/pull/27

[jakirkham]

Think a different strategy is needed for using the compiler in the test phase. When just using compiler during the test phase, conda-build only considers the build_platform (where the build is being run). It does not know the target_platform. So we need to provide this information another way. Think we need to adapt this logic like so:

    test:
      requires:
        - {{ c_compiler }}_{{ target_platform }} {{ c_compiler_version }}.*
        - {{ cxx_compiler }}_{{ target_platform }} {{ cxx_compiler_version }}.*
        - {{ cuda_compiler }}_{{ target_platform }} {{ cuda_compiler_version }}.*

[isuruf]

There's no difference in the two ways you mentioned.

[jakirkham]

Ok what alternative would you propose?

[isuruf]

Your analysis is wrong. The issue is not about build_platform. It's something else.

[carterbox]

I agree with isuruf. The logs show that, for example, in the build targeting aarch64, the following packages are installed in the test environment.

    cuda-cccl_linux-aarch64:                   12.0.90-h579c4fd_1      conda-forge
    cuda-cudart:                               12.0.107-hac28a21_6     conda-forge
    cuda-cudart-dev:                           12.0.107-hac28a21_6     conda-forge
    cuda-cudart-dev_linux-aarch64:             12.0.107-hac28a21_6     conda-forge
    cuda-cudart-static:                        12.0.107-hac28a21_6     conda-forge
    cuda-cudart-static_linux-aarch64:          12.0.107-hac28a21_6     conda-forge
    cuda-cudart_linux-aarch64:                 12.0.107-hac28a21_6     conda-forge
    cuda-driver-dev:                           12.0.107-hac28a21_6     conda-forge
    cuda-driver-dev_linux-aarch64:             12.0.107-hac28a21_6     conda-forge
    cuda-nvcc-dev_linux-aarch64:               12.0.76-h579c4fd_0      conda-forge
    cuda-nvcc-impl:                            12.0.76-hac28a21_0      conda-forge
    cuda-nvcc-tools:                           12.0.76-hac28a21_0      conda-forge
    cuda-nvcc_linux-aarch64:                   12.0.76-h028b88b_10     conda-forge
    cuda-version:                              12.0-hffde075_2         conda-forge

https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=771939&view=logs&j=bb1c2637-64c6-57bd-9ea6-93823b2df951&t=350df31b-3291-5209-0bb7-031395f0baa1&l=1293

[jakirkham]

JFYI we have added Windows support to the CUDA 12 migrator

This was done in PR: https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/4799

Also this is the last change we plan to make to this migrator

It follows recent work adding and testing CUDA 12 Windows packages

For any feedstocks that support Windows (and were already migrated to CUDA 12), the bot will submit a new CUDA 12 migration PR for Windows support

Any feedstocks that have a CUDA 12 migration PR open, but would like to add CUDA 12 for Windows, please ask the bot to re-run. The bot will then close out the CUDA 12 migration PR and add a new one with Windows included

For feedstocks still waiting to be migrated, no action is needed

Similarly for feedstocks that are Linux only and have already added CUDA 12, there is nothing to do. If the bot opens a migration PR, please feel free to merge or close as you see fit. The bot will mark it done either way

Please let us know if you have any questions 🙂

Thanks everyone! 🙏

[h-vetinari]

Any feedstocks that have a CUDA 12 migration PR open, but would like to add CUDA 12 for Windows, please ask the bot to re-run.

AFAIR, bumping the migration number is going to lead to a new PR even if one is already open.

[dhirschfeld]

I'm particularly keen on seeing jax on cuda12 but it seems the CI takes too long. I'm guessing there's no general-purpose solution yet for packages which exceed the allowed build times?

---

xref: https://github.com/conda-forge/jaxlib-feedstock/pull/207#issuecomment-1783748917

[jakirkham]

Any feedstocks that have a CUDA 12 migration PR open, but would like to add CUDA 12 for Windows, please ask the bot to re-run.

AFAIR, bumping the migration number is going to lead to a new PR even if one is already open.

Agree it should, but mentioned it just in case the bot ran into an issue doing that

Edit: Which I've seen a few times now already

[jakirkham]

An update here. So far we have added CUDA 12.0.0. We are now in the process of adding CUDA 12.x releases (with x > 0). In particular we are setting up CUDA 12.1.1 ( https://github.com/conda-forge/cuda-feedstock/issues/11 )

If users want to pin tightly to a specific CUDA 12.x version during the build, would recommend adding cuda-version in the relevant requirements sections. For example to constrain during the build (though not necessarily at runtime)...

requirements:
  build:
    ...
    - {{ compiler('cuda') }}                    # [(cuda_compiler or "None") != "None"]
  host:
    ...
    - cuda-version {{ cuda_compiler_version }}  # [(cuda_compiler_version or "None") != "None"]
  run:
    ...

[elgalu]

Hi, any updates?

[leofang]

Hi, any updates?

Update:

• CUDA 12.1.1: Released (https://github.com/conda-forge/cuda-feedstock/issues/11)
• CUDA 12.2.2: Released (https://github.com/conda-forge/cuda-feedstock/issues/13)
• CUDA 12.3.2: In progress (https://github.com/conda-forge/cuda-feedstock/issues/15)

[elgalu]

Thanks @leofang , do by chance know why https://anaconda.org/anaconda/cudatoolkit is still at 11.8 Last upload: 8 months or whom to contact?

[leofang]

Packages on the anaconda channel are owned and maintained by Anaconda folks, so usually you'll need to reach out to them (https://github.com/AnacondaRecipes/cudatoolkit-feedstock). In this particular case, @aterrel or @jakirkham might also know about their update plan.