[BanyanDB] Implement Read-Only Mode When Disk Usage Exceeds Watermark

[hanahmily]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

As disk usage approaches full capacity, data corruption and system failure risk increases. BanyanDB can enter read-only mode by setting a threshold to maintain data integrity and allow for administrative intervention.

1. Define a Disk Usage Watermark:

   • Allow users to configure a disk usage percentage (e.g., 80%) as a watermark.

2. Monitoring Mechanism:

   • Implement a background process to monitor disk usage metrics at regular intervals.
   • When the watermark is exceeded, the process triggers a switch to read-only mode.

3. Read-Only Mode Activation:

   • Disable all write operations while allowing read queries.
   • Logs and metrics will record this transition to alarm users.

4. Recovery and Normalization:

   • Once disk usage falls below a safe threshold, allow the system to return to normal operations.
   • Provide manual override options for administrators.

Use case

No response

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• [ ] Yes I am willing to submit a pull request on my own!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[wu-sheng]

Read-Only Mode Activation:

Besides you added, we need this as a new response code to notify OAP to affect OAP self-obs metrics as well.