dev dependencies with conda

[akhanf]

With hippunfold installed via conda, we now don't get the dev dependencies (e.g. snakefmt, isort, black) for linting and code quality.. What I've been doing now is using both conda and poetry, which is a bit cumbersome (switching between envs)..

@Dhananjhay do you know if there is a way to specify dev dependencies for conda packages? So one could enable them with a flag when using conda install hippunfold?

[Dhananjhay]

I don't think there exists a mechanism which is analogous to [tool.poetry.dev-dependencies] section in pyproject.toml in Conda . But what we can do is create another lightweight conda package, i.e., hippunfold-dev that will contain the dev-dependencies (e.g. snakefmt, isort, black) and instead of adding a flag, users would install both the packages from a single command conda install hippunfold hippunfold-dev -c bioconda. We can always pile the dev-dependencies on the existing hippunfold conda package but that'd make it bloated and might lead to some dependencies conflicts. What do you think?

[akhanf]

Yes that seems like an excellent workaround! Actually, could have hippunfold-dev that has the dev dependencies, along with hippunfold as dependencies, then could install with conda install hippunfold-dev right?

[Dhananjhay]

That's even better! On it right away!

[jordandekraker]

I think this is related - I currently have no good way to execute hippunfold (dev-v2.0.0)

This change broke my poetry and pip installs https://github.com/khanlab/hippunfold/pull/364, specifically the addition of lib/utils.py

I can install miniconda > snakemake > snakebids > ./hippunfold/run.py, but when I inculde --use-singularity I get the following error:

CreateCondaEnvironmentException:
Conda must be version 24.7.1 or later, found version 4.10.3. Please update conda to the latest version. Note that you can also install conda into the snakemake environment without modifying your main conda installation.

My conda is fully up-to-date, but 4.10.3 is the version inside hippunfold_deps.sif

When I run without the container, I get errors in wb_command since I don't have the latest version compiled locally.

Some help would be greatly appreciated!

[akhanf]

When you run in conda, don't use --use-singularity, instead use --use-conda

Best, Ali

---

From: Jordan DeKraker @.> Sent: Friday, February 7, 2025 2:14:01 PM To: khanlab/hippunfold @.> Cc: Ali Khan @.>; Author @.> Subject: Re: [khanlab/hippunfold] dev dependencies with conda (Issue #368)

I think this is related - I currently have no good way to execute hippunfold (dev-v2.0.0)

This change broke my poetry and pip installs #364https://github.com/khanlab/hippunfold/pull/364, specifically the addition of lib/utils.py

I can install miniconda > snakemake > snakebids > ./hippunfold/run.py, but when I inculde --use-singularity I get the following error:

CreateCondaEnvironmentException: Conda must be version 24.7.1 or later, found version 4.10.3. Please update conda to the latest version. Note that you can also install conda into the snakemake environment without modifying your main conda installation.

My conda is fully up-to-date, but 4.10.3 is the version inside hippunfold_deps.sif

When I run without the container, I get errors in wb_command since I don't have the latest version compiled locally.

Some help would be greatly appreciated!

— Reply to this email directly, view it on GitHubhttps://github.com/khanlab/hippunfold/issues/368#issuecomment-2643801586, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACXV2XJOIW4TCVJL4TJEBCL2OUAXTAVCNFSM6AAAAABWWCBCPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNBTHAYDCNJYGY. You are receiving this because you authored the thread.Message ID: @.***>

[jordandekraker]

Thank you!

Unfortunately I now have a new environment-related Error in rule laynii_layers_equidist

LN2_LAYERS: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by LN2_LAYERS)

[akhanf]

Thank you!

Unfortunately I now have a new environment-related Error in rule laynii_layers_equidist

LN2_LAYERS: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by LN2_LAYERS)

Ah I haven't seen that before -- maybe that's something maybe @Dhananjhay can debug? As a workaround can you use singularity? (ie omit the --use-conda flag and replace with --use-singularity) -- that should still work as it had before..

[Dhananjhay]

That's weird, I've never seen that before either. LAYNII has been tested on conda-forge's CI pipeline (https://github.com/conda-forge/staged-recipes/pull/28886) and locally on a linux system. Can you please share the command that you are running so I can reproduce this error, thanks!

[jordandekraker]

./hippunfold/hippunfold/run.py multihist7_BIDS multihist7_hippunfold participant --modality manualseg --path-manualseg multihist7_BIDS/sub-{subject}/anat/sub-{subject}_hemi-{hemi}_100um_dseg.nii.gz --keep-going --use-conda --cores 24 --exclude-participant bbhist2

there's not too much special going on here, but if you want to try with similar data you can use this (in manualsegs/)

i'll look around to see if there's something weird about my environment

[Dhananjhay]

Seems like it's due to the version of glibc against which I built and compiled LAYNII (https://stackoverflow.com/questions/70019660/including-glibc-in-a-conda-package); on CBS it's 2.31. @jordandekraker can you check what version you have installed on your system ldd --version. ~~Anyways, I'll try to downgrade the version to support a broader range of systems~~. Thanks for bringing this to my attention!

UPDATE: There's an open PR to submit LAYNII on conda-forge and in this case it's built against many versions of glibc

INFO :: The inputs making up the hashes for the built packages are as follows:
{
  "laynii-2.7.0-h629725b_0.conda": {
    "recipe": {
      "__glibc": "__glibc >=2.17,<3.0.a0",
      "c_stdlib": "sysroot",
      "c_stdlib_version": "2.17",
      "channel_targets": "conda-forge main",
      "cxx_compiler": "gxx",
      "cxx_compiler_version": "13",
      "target_platform": "linux-64",
      "zlib": "1"
    }
  }
}

So, I'm guessing once LAYNII can be pulled from conda-forge instead of our custom khanlab conda repo, you wouldn't face any issues with glibc.

[jordandekraker]

ldd --version:

ldd (Ubuntu GLIBC 2.27-3ubuntu1.6) 2.27
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

I've been meaning to upgrade to the latest ubuntu, perhaps this would be a good time

[Dhananjhay]

Ahh yes indeed, glibc's version on your system is v2.27 which is less than what LAYNII requires v2.29. Either you can wait for LAYNII to be available on conda-forge or you can update your os to latest ubuntu. If the latter seems like a headache I'd suggest using --use-singularity in the meantime, LAYNII's PR is under review and should be approved anytime now!

[jordandekraker]

Yup, that makes sense. Our sysadmin has been wanting to upgrade my ubuntu for a while now anyway, so we're doing that right now. Thanks for identifying the issue!

[akhanf]

@Dhananjhay did you end up making something like hippunfold-dev for conda? An alternative could just be to include a conda env file in the hippunfold root dir that we could use to set-up dev dependencies..

[Dhananjhay]

@Dhananjhay did you end up making something like hippunfold-dev for conda? An alternative could just be to include a conda env file in the hippunfold root dir that we could use to set-up dev dependencies..

Yep, it's available on bioconda channel hippunfold-dev! But I see your point. I believe only the users who would want to contribute to hippunfold would need to install the dev dependencies. In that case, it'd be much nicer to have a conda env file in the root dir which would create a new virtual env and then install snakebids and relevant dev dependencies in it. I guess hippunfold-dev might not be super useful to people who would want to contribute and hack locally.

[Dhananjhay]

@akhanf While I was writing documentation for downloading and running hippunfold with conda, I experimented with this alternate approach (PR #438) and it seems to be working well!

[Dhananjhay]

This has been resolved now.