cloud-pipeline
cloud-pipeline copied to clipboard
Allow to skip certain files/folders when applying quotas to the filesystems
Background Certain apps (like RStudio) use home directories, which are EFS volumes, to store session data. When the session is suspended - all the current environment is dumped into the home directory (it is common to have 50Gb+ session snapshot). If there are quotas set for the home directory - it gets reached quickly and the users are not aware of the reasons. We need to skip some of the "well-known" paths when indexing the EFS quotas to mitigate this issue.
Approach
- Introduce a preference to list the paths, which shall be skipped during filesystem volume computation
- When the size is computed we need to store two values:
- "Real" size - same as a current behavior, all the files volume
- "Effective" size - exclude the the files/folders, which fall into the preference
- Quotas shall be now applied based on the "Effective" size. "Real" size shall be used for billing reports.
- API shall expose both sizes in the data storage details, "Real" size is provided for the information.
- GUI shall display both sizes. "Effective" shall be shown by default. "Real" size shall be displayed in the brackets or in the tooltip.
- Preference shall support setting the list of "skipped" folders/files for a single storage or a list of storages. Each storage entry shall support basic "wildcards" (as we have e.g. for the allowed instance types). So it is possible to defined an "absolute" storage name (e.g.
my_storage1) or a set of storages in a single shot (e.g.my_storage*or even all storages with*).
Shall be backported to release/0.16
@sidoruka GUI part implemented (4d64e609197e6c5eeea2a773cd42043395075972)
@sidoruka backported to release/0.16 (595515d0f2a4750acd061988423f915690bda149)
@mzueva @Wedds please implement "wildcards" for the storageName field. See last point from the original issue.
@mzueva @Wedds please implement "wildcards" for the
storageNamefield. See last point from the original issue.
Done and backported to release/0.16
Docs were added via #3228 and located here.