Using Coiled with Nebari when pip packages are included in the environment

[maxrjones]

Context

We have successfully used Coiled clusters via Nebari in the past. However, we recently found that the disabling of pip in the later releases of Nebari causes issues when there are pip packages included in the conda environment. coiled.cluster() works fine when only conda packages are included in the environment. I expect the best solution would be to pass the docker image from conda-store into coiled.cluster(..., container="..."), but I am not sure how to enable Coiled to access the image.

More specifically, the issue arises in the creating wheels step of package sync:

import coiled
cluster = coiled.Cluster(n_workers=2)

╭───────────── Package Sync for carbonplan ─────────────╮
│ Fetching latest package priorities ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Scanning 727 conda packages        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:02 │
│ Scanning 425 python packages       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:02 │
│ Running pip check                  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:01 │
│ Validating environment             ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:01 │
│ Creating wheel for altair          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
╰──────────────────────────────────────────────────────────────────────────────╯

---Wheel Build Log for altair---
ERROR: Could not find an activated virtualenv (required).

Value and/or benefit

The combination of Nebari + Coiled provides users with the benefits of Nebari for persistent JupyterHub access and Coiled for the scaling and monitoring of Dask clusters.

Anything else?

No response

[dharhas]

For context.

We are setting PIP_REQUIRE_VIRTUALENV to True as an env variable to avoid pip from installing things in the .local folder. conda environments are placed in a read-only folder so pip cannot install into the parent environment and puts the packages in .local instead. This can break all python environments on Nebari which is why it is now disabled.

ERROR: Could not find an activated virtualenv (required). <- comes from the flag being active.

[jrbourbeau]

For reference, we just pushed out a new coiled release that should fix things even with the recent PIP_REQUIRE_VIRTUALENV=True change in nebari. @maxrjones mentioned he would try it out when he gets a chance

[maxrjones]

Thank you for your prompt help with this issue! I can confirm that the issue is resolved after specifying coiled==1.16.0 within the pip section of the conda environment file 🎉