[ENH] - Add new node group for conda store workers

[Adam-D-Lewis]

Feature description

We are looking forward to the conda worker scaling in https://github.com/nebari-dev/nebari/pull/2384. After it's merged, we want to move the conda store workers to it's own node group so the pods on the general node group don't exceed the general node group memory.

A few specifics:

• We want to add a new node group for most conda store workers.
• We want to keep 1 conda store worker on the general node group so that conda store builds are speedy (may require increasing size of general node group nodes). This should still scale down to 0 since conda store worker seems to have a memory leak issue see here.
• We want to set the threading to 1 (currently 4) on each of the conda store worker pods to keep things easy to reason about
• We need to get an idea of memory usage for an average conda store build so we can size the conda store worker nodes appropriately. Update: In very loose testing, memory usage doesn't seem to be more than a few GiB, it seems like it's more the memory leak issue mentioned above.

Value and/or benefit

Anything else?

No response

[pt247]

A few specifics:

• We want to add a new node group for most conda store workers.
• We want to keep 1 conda store worker on the general node group so that conda store builds are speedy (may require increasing size of general node group nodes)

Pre-KEDA merge was the default behaviour. And we can revert it. An Keda.ScaledObject can have multiple triggers. Each trigger can have its own podSelector.

• We want to set the threading to 1 (currently 4) on each of the conda store pods to keep things easy to reason about

This is part of KEDA PR.

• We need to get an idea of memory usage for an average conda store build so we can size the conda store worker nodes appropriately.

There is a way we can plot memory usage using Grafana. Last I checked, a sci-kit-learn + pandas build took the better part of 7Gigs. I wonder if we can get usage from our client deployments or ask the conda-store team.

[kcpevey]

Here is a pretty big environment which is based on one of our client needs (a few private packages removed):

channels:
  - conda-forge
  - bokeh
dependencies:
  - python=3.10
  - panel
  - ipykernel
  - ipywidgets
  - ipywidgets_bokeh
  - holoviews
  - openjdk=17.0.9
  - pyspark
  - findspark
  - jhsingle-native-proxy>=0.8.2
  - bokeh-root-cmd>=0.1.2
  - nbconvert
  - nrtk
  - xaitk-saliency
  - maite
  - daml
  - pytorch
  - loguru
  - pytest
  - pyright
  - torchmetrics
  - torchvision
  - multiprocess
  - smqtk-detection
variables: {}

This took 37 minutes to solve on nebari

[Adam-D-Lewis]

Brought up in June 4 community meeting, potentially use node types with NVME storage (e.g. i3en on AWS) as swap to accomodate conda store solves with high memory usage in a cost effective way.

[dharhas]

NVME storage as swap to accomodate conda store solves with high memory usage in a cost effective way

I think these volumes might be more expensive than what we do now.

[Adam-D-Lewis]

I think these volumes might be more expensive than what we do now.

The alternative I'm comparing against is using conda-store-worker nodes (and pods on those nodes) with enough memory allocated to handle the largest imaginable conda builds. However, most builds aren't going to need the the extra memory so that would waste resources. I assume having some NVMe swap available for the odd case where we need extra RAM would be cheaper than that, but I agree we should check to know for certain. Additionally, we want to make sure this is a cross cloud solution.

My understanding is that NVMe swap is still much slower than RAM, so we'll need to test that this is reasonable.