Grace period for the FS quotas

[sidoruka]

Background At the moment, FS quotas are applied immediately as they are reached. While this is totally correct - users may face the compute workloads to stop. To address this - we need to implement a grace period for the quotas restrictions to become effective.

Approach

• Grace period shall be configurable for both states: DISABLE_MOUNT and READ_ONLY. It's ok to introduce two separate preference or a single one with the JSON spec
• When the data storage volume exceeds a threshold - we shall not move it's state to DISABLE_MOUNT or READ_ONLY until the grace period exceeds
• If a certain state does not have a grace period - everything shall behave as it is now (immediate state switch)

Example

• E.g. we have a storage with 10Gb volume and the following quota setup:
  • 20Gb: Disable mount
• Grace periods are set as:
  • Disable mount: 24h
• User uploads 15Gb file and this information gets into the index
• "Effective" storage size (#2399) exceeds the Disable mount quota
• The quota monitor performs the following actions:
  • Sends a notification to the user, that a quota has been reached, but the state is still ACTIVE
  • Does NOT change the state of the storage
• Once 24 hours are elapsed and the size of the storage is still 25Gb:
  • Storage state is set to DISABLE_MOUNT
  • User is notified about that event

[sidoruka]

Please backport to release/0.16

[mzueva]

@sidoruka server backported to release/0.16 41bd8cf16626ec0340587d3f13f3ad6e6b6ce7b5

[maryvictol]

Test cases were created by https://github.com/epam/cloud-pipeline/pull/2549 and located here.

[NShaforostov]

Docs were added via https://github.com/epam/cloud-pipeline/pull/3228 and located here.