Support conda (python) package manager

[rarkins]

https://conda.io/docs/

[meg-hegde]

Hi, does Renovate now support conda?

[meg-hegde]

Hi, just wondering whether there are plans to add conda support soon? Alternatively, shall I try adding it using the instructions here: https://github.com/renovatebot/renovate/blob/master/docs/development/adding-a-package-manager.md?

[rarkins]

No plans, and a PR would be very welcome! I updated the doc just now to make sure it's current.

[meg-hegde]

Thank you for updating the docs - I'll give this a go when I have some time :)

[gerbenoostra]

#6969 duplicated this, thus closed it. The relevant comments from there: What would you like Renovate to be able to do?

To also verify python package versions in conda environment files (environment.yml)

Did you already have any implementation ideas? no

Are there any workarounds or alternative ideas you've tried to avoid needing this feature?

Conda environments can also include pip requirememnts, a workaround is to put those in a separate txt file, and have renovatebot check those. environment.yml would then be:

dependencies:
- python=3.7
- jupyter
- pip
- pip:
  - -rrequirements.txt

This workaround however does not work for the conda packages (like the python=3.7 here, and any conda packages installed, like jupyter in this case)

Is this a feature you'd be interested in implementing yourself? maybe

** Related features** Relates to #931 , but that only implemented pip dependencies.

[rarkins]

It would be helpful if anyone can provide some public repo examples that can be tested against, as well as clarifications on file naming / file syntax. For example should we match against every environment.yml file in the repo?

[padmick]

From their docs https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file they reference it against environment.yml files. I know internally we use different names for our different projects (IE we name our env the same codename as the project so titan.yml ect) but if that is an issue we can look at changing them back to the conda default name.

[davidspek]

I'm also interested in this as the PyTorch images (and the jupyter-stack ones) use conda.

[AndreaGiardini]

We could give it a shot, starting from the datasource. I am not very familiar with typescript but I could give it a go.

The docs in the datasource require a function called ``getReleases` with input:

`lookupName`: the package's full name including scope if present (e.g. @foo/bar)
`registryUrls`: an array of registry Urls to try

If I understood it correctly, for something like https://anaconda.org/conda-forge/proj/ we will have:

lookupName -> conda-forge/proj registryUrl -> https://anaconda.org/

Am I missing something?

[davidspek]

I would just like to make a note that the conda-forge channel would be important to have support for (also in terms of ToS of the regular conda channel).

[rarkins]

I agree that starting with the datasource first makes best sense.

Importantly, note that Renovate doesn't yet have the concept of "platform" for datasources but it looks like that might be necessary for conda packages?

[mathbunnyru]

Jupyter docker images would also greatly benefit from this feature. https://github.com/jupyter/docker-stacks/issues/1153

Right now, we're updating our dependencies manually, and it would be great to get rid of this maintenance burden.

[morremeyer]

Hey everyone, Anaconda engineer here. We've assembled a small group of engineers that is looking into adding functionality for conda over time.

Please note that none of us is working on this full time right now, but work will be done over time.

I've started implementing a datasource for conda in #14257, any help with my testing issue and feedback in general is very welcome!

[mdehollander]

Hey everyone, Anaconda engineer here. We've assembled a small group of engineers that is looking into adding functionality for conda over time.

Please note that none of us is working on this full time right now, but work will be done over time.

I've started implementing a datasource for conda in #14257, any help with my testing issue and feedback in general is very welcome!

Thanks for starting the work on this! I am happy to start testing, but I am not sure how since I am new to renovate. I gave it a try with adding a renovate.json to my repo (https://github.com/mdehollander/orochi/blob/master/renovate.json). The conda environment files are the folder src/envs/. But I did not manage to set it up correctly since I get this error:

validationMessage": "Invalid configuration option: conda, The following managers configured in enabledManagers are not supported: \"conda\"",

What is the best way to test and configure this? Or I am too early :smiley_cat:

[morremeyer]

@mdehollander You're not too early, but there's no manager implemented yet, just a datasource. I'll take a first shot at a manager next Friday.

You can check the datasource documentation at https://docs.renovatebot.com/modules/datasource/#conda-datasource.

If you want to use it right now, here's an example for how to do so. In your environment.yml, add a comment that annotates the line for renovate:

name: your-project
channels:
  - defaults
dependencies:
  - pytest
  - pytest-cov
  - coverage
  # renovate datasource=conda depName=main/yapf
  - yapf==0.31.0

This annotates the yapf package for the regexManager and will query the main channel for the versions. You can then use the regexManager with the following configuration:

{
  "reviewersFromCodeOwners": "true",
  "regexManagers": [
    {
      "description": "Upgrade conde dependencies",
      "fileMatch": [
        "(^|/)environment.yml$"
      ],
      "matchStrings": [
        "# renovate datasource=conda\\sdepName=(?<depName>.*?)\\s+- [a-z0-9]+==\"?(?<currentValue>.*)\"?"
      ],
      "datasourceTemplate": "conda"
    }
  ]
}

The job of the package manager is to do the discovery/annotation that I show above automatically so that you don’t need any configuration in the default case.

[mdehollander]

Thanks for the extra information and the example config. With this I managed to get a PR triggered for an update of a conda environment. From the logs:

DEBUG: Matched 33 file(s) for manager regex: src/envs/amos.yaml, src/envs/antismash.yaml, src/envs/bamm.yaml, src/envs/bbmap.yaml, src/envs/bedtools.yaml, src/envs/bigscape.yaml, src/envs/bigslice.yaml, src/envs/bwa.yaml, src/envs/cat.yaml, src/envs/checkm.yaml, src/envs/concoct.yaml, src/envs/coverm.yaml, src/envs/dastool.yaml, src/envs/fraggenescan.yaml, src/envs/groopm.yaml, src/envs/khmer.yaml, src/envs/kraken.yaml, src/envs/mash.yaml, src/envs/megahit.yaml, src/envs/metabat.yaml, src/envs/minimap2.yaml, src/envs/mmgenome.yaml, src/envs/mmgenome_prepare.yaml, src/envs/pigz.yaml, src/envs/prodigal.yaml, src/envs/quast.yaml, src/envs/report.yaml, src/envs/samtools.yaml, src/envs/seqtk.yaml, src/envs/spades.yaml, src/envs/tree.yaml, src/envs/vamb.yaml, src/envs/vsearch.yaml

DEBUG: 2 flattened updates found: bioconda/spades, bioconda/vsearch
DEBUG: Returning 2 branch(es)
DEBUG: Fetching changelog: https://github.com/ablab/spades (3.14.0 -> 3.15.4)

And 2 PRs for 2 packages that I enabled: https://github.com/mdehollander/orochi/pulls

To get the environment files in subfolders recognized I changed the regular expression in the config to:

      "fileMatch": [
        "^(?:src/envs/)?\\w+\\.yaml$"
      ],

Looks very promising! Thanks! Looking forward for a manager :)

[mdehollander]

@morremeyer I am wondering if you managed to work on automatic discovery of conda packages via the package manager. That would make the use on existing installation easier, because you don't need to annotate the env files :)

[morremeyer]

Unfortunately, I didn’t get around to it yet as I needed to prioritize other topics. I’m estimating to get some work in for this in about 3 weeks.

[mdehollander]

Hi, any progress on conda support via the package manager? I would like to mention/promote this in a presentation and if there have been any updates it would be nice to know. Thanks.

[PhilipAbed]

we have the Versioning and Datasource for python(also for conda datasource now), we just need to check for environment.yml file and extract dependencies and it should work with renovate normally

In fact we can do this using regex managers for now i think, if not we will need to start with the datasource as Rarkins suggested

[morremeyer]

Hey everyone, sorry for not updating this sooner. I looked into the conda versioning spec a while ago and have a short write-up of what my plans would be, but somehow forgot to post that here.

I'm out of office right now and set a reminder to update this issue once I'm back, which should be by the end of this week.

[morremeyer]

@PhilipAbed I'm not sure if the python datasource supports the conda versioning spec.

Everyone, here's what I thought I had posted back in May when I last got to work on this (time really flies currently):

We worked on this over the last few days, so here is a summary and the plan for how I’m thinking about proceeding here.

There is no working manager yet that could be tested as the focus was on two things:

• Ensure that we only create correct updates
• Start simple, build support for more complex cases later

As the conda MatchSpec enables complex definitions of package versions, it’s vital to parse them correctly according to the specification so that we don’t create wrong updates.

A MatchSpec can match on practically every attribute of a package, so we need to focus on the common cases that can be correctly implemented without hacking the renovate core.

I discussed a lot of ideas with and got input by @rarkins, big thank you for that!

Here’s what is planned to be supported in the first iteration of a conda manager:

• Specifications of dependencies that include channel, subdir, name, version. Check the MatchSpec documentation linked above for details on the fields.

The first iteration will have at least the following limitations:

• No support for channel priority, it will behave like conda with channel_priority: disabled
• Dependencies with any of the following fields specified will be ignored:
• namespace (this is not used yet in conda, but ignoring it will prevent wrong updates)
• build will be ignored
• any keyword argument will be ignored
• The key pip in the dependencies will be ignored if it is a list
• No support for the nonstandard keys introduced by conda-lock.

This means that specifications like the following should be supported:

• conda-forge::some-package=1.0
• numpy=1.2.3

More complex cases can then be added over time.

I will be working on this the week after next (likely 2022-08-10 and 2022-08-11), I'm happy about any feedback here!

EDIT for additional resources:

• conda version specification
• supported version strings

[PhilipAbed]

Thanks for the update, looks promising! good work

[srilman]

What is the status of the PR? I see that the last update was a couple of months ago. Would be happy to help get it to the finish line!

[viceice]

there's no more work from PR author after suggestions. so he's maybe no longer working on it. feel free to take it over by create a new PR with requested changes.

[srilman]

Sounds good, thanks!

[morremeyer]

Hey everyone!

Sorry for the silence over the last months. We still have this on our agenda, but didn't get to work on it in recent weeks.

Working on implementing the manager, we realized that we will first need to implement conda versioning to implement a manager that works reliably and consistently.

We definitely have not abandoned this, but as stated initially we cannot give guarantees for progress here.

If anyone else wants to contribute to this, we're more than happy to put in our feedback for PRs.

[pavelzw]

Maybe it could make sense looking into pixi. They use the conda universe but in a somewhat structured way. Pixi projects always use a pixi.toml and have a pixi.lock which contain the lockfile. Somewhat similar to poetry but for conda. Probably it's easier to implement this into dependabot since this has a clearly defined project structure.

[viceice]

@pavelzw please open a new discussion and reference this issue.