Creating conda environment fails with non-descriptive error message

[sebschmi]

When setting up miniconda I get the following last lines in the log:

Creating conda environment from yaml file...
  /usr/share/miniconda/condabin/conda env create --file environment.yml --name practical-omnitigs
  Collecting package metadata (repodata.json): ...working... done
  Solving environment: ...working... ::error::The process '/usr/share/miniconda/condabin/conda' failed with exit code null

This makes the job fail. I searched for error with the search function in the log, but got no other error message. Is there a way to get a more descriptive error message?

Here is the job that fails.

[jaimergp]

I am able to install the environment file locally with mamba, so that's definitely solvable. Maybe it was just a sporadic error. Try re-running the job?

[jaimergp]

Hm, maybe not. Just took a look at your PR and it has been happening in previous attempts.

A couple of things:

• This line should match the name in environment.yml
• The environment is not complex by any means, so I don't think you need this, but try using mamba-version: "*" to enable mamba solving.

[sebschmi]

Hi, thanks for the quick reaction. After changing the name of the environment as you stated, everything works now.

I am confused though if that was some caching problem or the problem with the wrong name, as this error started occurring after changing only the dependencies in the environment.yml.

I guess it would be nice if miniconda gave a more descriptive error message.

[goanpeca]

Should we close this @jaimergp ?

[jaimergp]

I think we should check whether we can give a more descriptive message for this error, but I am not even sure this code path would be reachable after the refactor that is happening now. We should leave a note somewhere, at least.

[zjffdu]

I hit a similar issue, could anyone help take a look at it ? Here's my job link https://github.com/zjffdu/zeppelin/runs/4295470489?check_suite_focus=true#step:7:295 And this is my github action definition. https://github.com/zjffdu/zeppelin/blob/ZEPPELIN-5576/.github/workflows/core.yml#L197

[jaimergp]

Can you check whether using Mamba instead of Conda helps print a more helpful output? This might be a system error (out of memory or similar).

[zjffdu]

@jaimergp I enabled using Mamba, but still no error in the build output Here's my github action: https://github.com/zjffdu/zeppelin/blob/ZEPPELIN-5576/.github/workflows/core.yml#L110 This is the github action: https://github.com/zjffdu/zeppelin/runs/5213310585?check_suite_focus=true

[raphaelquast]

hey, I've started running into that same issue (nothing changed with the environment and it just started failing without any noticeable reason)

any updates on this? (I've updated the actions to use v2.1.1 but it didn't help)

[goanpeca]

Hi @raphaelquast, thanks for pinging, if you could provide some links to your workflows that fail we might have a better picture of what is going on.

Cheers!

[raphaelquast]

Hey, thanks for the response! Here's a link to one of the tests that kept failing during the installation without any descriptive info on what's going on: https://github.com/raphaelquast/EOmaps/runs/6589957054?check_suite_focus=true

I did some checking and I've managed to get my tests running again by switching over to using mamba, e.g. by adding the following lines:

- uses: conda-incubator/[email protected]
  with:
        ....
        .... 
        mamba-version: "*"
        channels: conda-forge
        channel-priority: true

My guess would be that the installation of the environment was terminated because solving it with conda took too long? (the env I'm using has a lot of rather complex dependencies to solve...)

[goanpeca]

Thanks for the input @raphaelquast !

Glad to hear mamba made things better for your use case. Now, would it be possible to have your dependencies list and the system you are running on to try to debug things locally?

That would help us a lot!

Cheers :)

[raphaelquast]

hey, of course that's possible! 🙂

I've used the following environment for testing: conda-yml for EOmaps

...and the tests on GitHub were running on "Ubuntu latest" (according to https://github.com/actions/virtual-environments this is currently 20.04)

[ychen-ot]

Thanks everyone for the help. I also ended up using the additional settings w/ enabling mamba to fix the issue.

[kelly-sovacool]

I'm getting this error when running the action on ubuntu-latest, but it works on macos-latest. I'd prefer to be able to use ubuntu because a later step in the job requires it.

In the actions yml:

- uses: conda-incubator/setup-miniconda@v2
  with:
    python-version: 3.11
    mamba-version: "*"
    channel-priority: true
    activate-environment: github-actions
    environment-file: workflow/envs/github-actions.yml

The conda env file (workflow/envs/github-actions.yml):

name: github-actions
channels:
  - conda-forge
  - bioconda
  - defaults
  - r
dependencies:
  - black=22.10.0
  - python=3.11.0
  - r-base=4.2.2
  - r-styler=1.8.1
  - r-usethis=2.1.6
  - snakefmt=0.8.0

[raphaelquast]

@kelly-sovacool I'm not associated with this repo, but since i read your comment...

you could try using mambaforge instead of conda+mamba.... it solved all issues for me... (and it's even faster) (like here: https://github.com/raphaelquast/EOmaps/commit/e7f4590e5f28054d345e8090146126a2dad1c63f)

[mattwthompson]

The root here here is usually something in conda itself and in my experience stems from the environment not being solvable at all.

Pinning all the way down to patch versions for all dependencies is not advisable in this ecosystem whereas it might be convention in others. Pins such as black=22.10.0 seem innocuous alone but regularly cause issues with larger environments. They're also probably not necessary, i.e. it's unlikely that a slightly newer or older versions of a given package actually gives behavior that's so different it must be forbidden

[kelly-sovacool]

The root here here is usually something in conda itself and in my experience stems from the environment not being solvable at all.

Pinning all the way down to patch versions for all dependencies is not advisable in this ecosystem whereas it might be convention in others. Pins such as black=22.10.0 seem innocuous alone but regularly cause issues with larger environments. They're also probably not necessary, i.e. it's unlikely that a slightly newer or older versions of a given package actually gives behavior that's so different it must be forbidden

This particular environment was solvable locally.

[kelly-sovacool]

@kelly-sovacool I'm not associated with this repo, but since i read your comment...

you could try using mambaforge instead of conda+mamba.... it solved all issues for me... (and it's even faster) (like here: raphaelquast/EOmaps@e7f4590)

This worked for me, thanks @raphaelquast!

New action yml:

- uses: conda-incubator/setup-miniconda@v2
  with:
    python-version: 3.11
    miniforge-variant: Mambaforge
    miniforge-version: latest
    activate-environment: github-actions
    environment-file: workflow/envs/github-actions.yml

[lumbric]

I'm having the same issue. In my case the job was running fine on 28th of December 2022, but it failed now after 3 weeks without notable change in the conda setup in between. So I guess something (conda forge repository, the setup-miniconda GH action, something on GH, ...) changed recently upstream. @raphaelquast's workaround fixed the issue on my side too.

[jaimergp]

I wonder if this is "just" a memory issue. The classic solver is known to hog a lot of resources trying to solve complex environments and/or explaining conflicts.

If macOS works but Linux doesn't it could be either:

• the packaging landscape being easier to traverse on macOS
• having more memory available on those runners.

One thing to try is to export an environment variable (at the global level) to increase verbosity:

env:
 CONDA_VERBOSITY: 3

And see what's going on.

The mamba solver is not only faster, but also way more efficient in terms of memory usage.

In terms of why it used to work but now it doesn't... conda-forge keeps growing and repodata gets bigger and more complex. We might be at a point where linux-64 is "too big for GHA runners". But this is just a guess and we need actual measurements of the memory usage. If anyone knows how to do that easily on GHA jobs, please let us know!

[amal-ghamdi]

I'm having the same issue. In my case the job was running fine on 28th of December 2022, but it failed now after 3 weeks without notable change in the conda setup in between. So I guess something (conda forge repository, the setup-miniconda GH action, something on GH, ...) changed recently upstream. @raphaelquast's workaround fixed the issue on my side too.

I am facing a similar situation, and @raphaelquast's workaround solution seems to work for me too 👍