[CELEBORN-1595] Support quota low watermark for checking quota available

[s0nskar]

What changes were proposed in this pull request?

Support quota low watermark for checking quota available. This will not allow new jobs to run on Celeborn if quota used is above lowWatermark.

Why are the changes needed?

Currently we allow jobs to run even if we're just about to breach quota limits. This is not ideal behaviour, ideally we should not allow any new jobs to run on Celeborn after certain threshold (called lowWatermark here). This will ensure current running jobs will use the quota and will not go way beyond quota usage.

I'll also follow up with a PR to throw CelebornIOException, if quota is breached.

Does this PR introduce any user-facing change?

NA

How was this patch tested?

UTs

[s0nskar]

IMO this configuration can be a dynamic configuration. Wdyt?

[s0nskar]

cc: @SteNicholas @waitinfuture @mridulm

[s0nskar]

Also, IMO quota should be a hard limit any application breaching quota should get a CelebornIOException. We can improve the application heartbeat to check for current quota.

[s0nskar]

ping @SteNicholas @waitinfuture

[s0nskar]

ping @FMX @SteNicholas @waitinfuture wdyt about this.

[github-actions[bot]]

This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions[bot]]

This issue was closed because it has been staled for 10 days with no activity.