cloud-pipeline icon indicating copy to clipboard operation
cloud-pipeline copied to clipboard

Allow to skip certain files/folders when applying quotas to the filesystems

Open sidoruka opened this issue 3 years ago • 6 comments

Background Certain apps (like RStudio) use home directories, which are EFS volumes, to store session data. When the session is suspended - all the current environment is dumped into the home directory (it is common to have 50Gb+ session snapshot). If there are quotas set for the home directory - it gets reached quickly and the users are not aware of the reasons. We need to skip some of the "well-known" paths when indexing the EFS quotas to mitigate this issue.

Approach

  • Introduce a preference to list the paths, which shall be skipped during filesystem volume computation
  • When the size is computed we need to store two values:
    • "Real" size - same as a current behavior, all the files volume
    • "Effective" size - exclude the the files/folders, which fall into the preference
  • Quotas shall be now applied based on the "Effective" size. "Real" size shall be used for billing reports.
  • API shall expose both sizes in the data storage details, "Real" size is provided for the information.
  • GUI shall display both sizes. "Effective" shall be shown by default. "Real" size shall be displayed in the brackets or in the tooltip.
  • Preference shall support setting the list of "skipped" folders/files for a single storage or a list of storages. Each storage entry shall support basic "wildcards" (as we have e.g. for the allowed instance types). So it is possible to defined an "absolute" storage name (e.g. my_storage1) or a set of storages in a single shot (e.g. my_storage* or even all storages with *).

sidoruka avatar Dec 13 '21 21:12 sidoruka

Shall be backported to release/0.16

sidoruka avatar Dec 14 '21 10:12 sidoruka

@sidoruka GUI part implemented (4d64e609197e6c5eeea2a773cd42043395075972)

rodichenko avatar Dec 20 '21 11:12 rodichenko

@sidoruka backported to release/0.16 (595515d0f2a4750acd061988423f915690bda149)

rodichenko avatar Dec 20 '21 11:12 rodichenko

@mzueva @Wedds please implement "wildcards" for the storageName field. See last point from the original issue.

sidoruka avatar Dec 21 '21 20:12 sidoruka

@mzueva @Wedds please implement "wildcards" for the storageName field. See last point from the original issue.

Done and backported to release/0.16

mzueva avatar Dec 22 '21 12:12 mzueva

Docs were added via #3228 and located here.

NShaforostov avatar Oct 17 '23 17:10 NShaforostov