[FEATURE]: Support Ephemeral Volumes, specifically emptyDir backed by Memory

[OOPMan]

Feature Request

Background / Motivation

K8S provides a very useful feature in the form of Ephemeral Volumes. These are useful for allocating temporary storage, including memory-backed RAM disk style volumes.

RAM disk volumes are very useful for various purposes, including mounting to /dev/shm to customize the amount of shared memory available to a Container which is in turn very useful when running PostgreSQL using Garden.

What should the user be able to do?

Define Volumes for Services on Container modules that use an emptyDir volume with it's medium set to Memory

Why do they want to do this? What problem does it solve?

This will allow me to use PostgreSQL within Garden without it (the database) exploding when I run any kind of query returning a large resultset

Suggested Implementation(s)

Add a new type of module for emptyDir with medium set to Memory volumes. Using an instance of an emptyDir module with a Service is the same as using a PVC module

How important is this feature for you/your team?

🌵 Not having this feature makes using Garden painful

[Orzelius]

Hello, just from my personal subjective experience: volumes and kubernetes do not work well together, they always seem to lead to weird behavior that's unpredictable and hard to debug. If you need a large prod-like database I'd avoid k8s all together and would think about using the terraform/pulumi providers to set up a real one with your cloud provider or use a staging shared database and have garden run some migrations with the exec module type.

This way you'll have less drift between your prod and dev env as both would be using the same real database and not something mocked up in k8s.

edit: could you expand a bit on your use case because I'd be interested to see if k8s-native databases make more sense in your scenario. Also, using garden kubernetes module type would allow you to deploy your own volume resource or you could add the emptyDir flag on the volume spec of whatever you're deploying.

[OOPMan]

@Orzelius You want another use-case? Okay, here is another one which is actually pretty significant IMHO.

There is a recommendation to run your Pods with a read-only root filesystem. This plays havoc with a lot of applications however as they expect at least /tmp to be writeable. So you either run with a RW root filesystem or you use a, you guessed it, Memory-backed emptyDir volume.

I realise I could use the K8S module type with Garden but, frankly, that just leaves me asking why I'm bothering with Garden in the first place. I already have a CI that builds images so I might as well just deploy standard manifests with images built by the CI.

I don't think finding reasons for me to not use Garden is a good response.

[OOPMan]

I'm sorry if my tone seems a bit rough but right now we only use Garden for development environments and part of this is due to the limitations it imposes.

Other teams in my company just use Docker-Compose in their development environment, our team prefers having a development environment that at least somewhat resembles the actual target deployment environment as it helps us pick up on issues that aren't immediately clear when using an alternative environment that bears minimal resemblance to a real one.

Dealing with the scenario where our development environment is unable to host a basic development database environment is just unhelpful and it's compounded by the fact that getting people to run things bare metal these days is an uphill battle. At some point in the last decade or so it feels like a lot of developers were convinced that they need to run everything inside a container even locally and getting them off that hill is like pulling teeth :-/

[edvald]

Hey @OOPMan. First off, I actually think the feature request makes sense and is well reasoned. I don't think @Orzelius meant to be dismissive.

From the other side, it's worth explaining that we tend to pause at adding more flags to the container module type, since basically the list of potentially useful flags on K8s Pod specs etc. kinda defies abstraction at times... So I think it's reasonable to raise the option to use the kubernetes type directly, along with a container module to do the image build, to see if that might be palatable.

We do that ourselves for our own development, and certainly still find plenty of value in Garden (obvious bias notwithstanding 😬 ). You can still do code syncing, tests, tasks, all that jazz, but you do in turn lose the cleaner and more limited config. We find that achieves the goal of being more production-like during dev nicely, but every case is different of course.

All that said, request is accepted, with the caveat that we might not go about it the suggested way. The services[].volumes[].module field was conceived to reference persistent volumes, so it's not quite a clean fit. Might be simpler to just add an inMemory or mode option to services[].volumes[].

Curious though: Why do you specifically need the Postgres volume to be in-memory as opposed to just ephemeral on disk? The latter is already possible, so I figured I'd ask.

[OOPMan]

@edvald Thanks for the response. I have considered combining the k8s and container type modules but unfortunately I've been so flooded with work that I haven't had a chance to dig into it. That said, it may well be something for my team to consider as it would, indeed, achieve an even closer fit between our dev and production environments.

In the context of Postgres, the specific reason I request support for in-memory was based on an assumption on my part that a disk-based volume mounted to /dev/shm would probably not perform the same way as a memory-based volume mounted to /dev/shm.

I should probably test using a disk-based volume but I suspect it would perform rather poorly and given the issue I've run into is specifically due to the pokey standard 64mb allocation to /dev/shm in a container being insufficient I suspect that using a disk-based volume would probably introduce a different kind of problem.

[edvald]

Gotcha. I mean yeah, I'm pretty sure RAM is going to beat disk on performance, couldn't say by how much in this particular context though. Guess you could find out by creating the volume on the container module and skipping both module and hostPath. In that case, a basic emptyDir volume is created.

Feel free to ping btw if you're getting into k8s+container combos. This example might help, but feels like we could do with more examples and docs on this...

[OOPMan]

@edvald Thanks for that example, I'll take a look at it :-)

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!

[OOPMan]

@edvald Any movement on this?

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!

[10ko]

Going forward, we'll suggest to use the kubernetes action type for anything very specific or when you need more flexibility. Closing this as not planned.