nebari-dev
[ENH] - Limitation of storage space per user
Feature description
I want to limit each user to 100 MB of storage space. What are my options for doing this?
Value and/or benefit
nebari as a person's paid platform means that we can’t assume everyone will “place nice” and share space appropriately. And we can’t just just delete people's data to make extra space. Limiting their access in the first place is the best option.
Anything else?
I'd like to see the ability to say "X user group gets 100 MB of space per user".
We need to think through how this will work in the shared folders as well.
cc @iameskild
Just thinking about the shared folder case - on big HPC systems, they implement a purge policy where anything older than X days is subject to purge. We'd probably need a warning system to keep users aware/happy. Not the best option, but as a last resort, might work.
I certainly think we could easily enforce per-user storage quotas by creating a pvc for each user with a specific size. This would also have the effect of allowing us custom storage for each user e.g. ssd vs. hdd storage. This is a low effort PR.
Like @iameskild said
it might be difficult to limit their storage in the shared directory given that everyone is technically the same user
I don't see any short term feature in the that would allow us to do this. Potentially how I see we could achieve this is to have a separate shared directory for each group and set a disk size for that. This would be non-trivial to implement (due to the storage type in kubernetes needing to be Read-Write-Many).
There are like you've said @kcpevey ways that we can nudge users to follow resource limits e.g. a purge policy or message to user that they are over the quota.
Currently every user's home directory is part of the same PVC/PV which makes it very difficult to limit the amount of storage a particular user has access to. As @costrouc mentioned, to get around this we will need to create a new PVC/PV for each user.
The desired behavior is the default on vanilla z2jh deployments; each user to gets their own PVC/PV so this we may simple need to modify this function.
However, to setup it up so that certain users have one sized PVC/PV and another set of users have another sized PVC/PV is a heavier lift. Assuming the size of the PVC/PV is determined by which Keycloak group the user belongs, some additional logic is needed to determine which group the user belongs to. This brings up a few questions:
- if a user is added to this new group (with larger PVC/PV), is there PVC/PV resized automatically?
- what about the reverse?
cc @kcpevey
Just to say glad to see this being raised, for my team I'd like to be able to set a storage quota specifically for each user, that would do the job for us. There was a bit of discussion of this previously over in https://github.com/nebari-dev/nebari/discussions/706
Summary of potential options based on continued internal discussions:
- Give all users the same amount of storage - no cleanup
- All users get their own PVC/PV for their home directory - relatively low effort (this is the default for z2jh)
- Every user gets the same amount, no ability to specify by user
- Total shared folder size specified only via config (not per user)
- Customize storage you want by user group (via PVC/PV)
- Give users, based on their group, different amounts of storage
- Challenges that will need to be worked out:
- if a user is added to this new group (with larger PVC/PV) the PVC/PV is not resized automatically
- We’d need to migrate the data to a new volume
- Total shared folder size specified only via config (not per user)
- Purge policy: user’s data deleted after N amount of time, may include warnings, etc
- Gold standard: Full linux permissions
- Would allow for customization at the individual user level - each user can have a different storage volume.
- This is the only option that will (theoretically) allow for limitation per user of the shared folder.
I want to return to the idea of full linux permissions (assuming volume size is still limited by user). I think that is indeed the gold standard and perhaps warrants an RFD. This would certainly make for a better user experience. I also wonder how it might tie in with keycloak (if at all).
For me I could see this as 2-3 days of work. But less than a week of work. This is this issue https://github.com/nebari-dev/nebari/issues/1549 roughly.
We would basically need to add a nebari option passed into the jupyterhub config to set the user home storage quota and if set nebari will not set home-pvc:
- https://github.com/nebari-dev/nebari/blob/develop/nebari/template/stages/07-kubernetes-services/jupyterhub.tf#L97
- https://github.com/nebari-dev/nebari/blob/develop/nebari/template/stages/07-kubern[…]/kubernetes/services/jupyterhub/files/jupyterhub/03-profiles.py
- within this function check is home-pvc is none and if so use a dynamic pv (default in jupyterhub) and set the capacity https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/main/jupyterhub/values.yaml#L378.