curve icon indicating copy to clipboard operation
curve copied to clipboard

【Feature】When the cluster capacity is almost full, make the cluster read only

Open liuminjian opened this issue 1 year ago • 9 comments

What problem does this PR solve?

Issue Number: #2561

Problem Summary: When the space of a single chunkserver of curvebs is insufficient, chunkserver will down directly

What is changed and how it works?

What's Changed:

1.Heartbeat reports disk full error and mds set copyset availflag false and set disk status error.
2.Copyset node leader set readonly when receive copyset availflag false from heartbeat.
3.If the disk becomes full while writing to the chunk file, the server return no space err and client hangs until space is freed up manually.

How it Works:

1.When the disk is full, the heartbeat uploads the disk status. MDS sets the disk status to error to prevent other copysets from migrating to this disk, and sets the copyset to be unavailable to avoid creating new space from these copysets.
2.When copyset status is unavailable, copysetnode will be set to readonly. when a new write request comes in, a read-only prompt will be returned.
3.If the disk becomes full while writing to the chunk file, the server return no space err and client hangs until space is freed up manually.

Side effects(Breaking backward compatibility? Performance regression?): Older versions of chunkserver need to add disk limit usage percentage configuration

Check List

  • [ ] Relevant documentation/comments is changed or added
  • [ ] I acknowledge that all my contributions will be made under the project's license

liuminjian avatar Nov 06 '23 10:11 liuminjian

cicheck

wuhongsong avatar Dec 01 '23 02:12 wuhongsong

cicheck

YunhuiChen avatar Dec 02 '23 03:12 YunhuiChen

cicheck

YunhuiChen avatar Dec 02 '23 03:12 YunhuiChen

cicheck

YunhuiChen avatar Dec 02 '23 04:12 YunhuiChen

cicheck

wu-hanqing avatar Dec 25 '23 06:12 wu-hanqing

cicheck

wu-hanqing avatar Dec 25 '23 08:12 wu-hanqing

cicheck

wu-hanqing avatar Dec 25 '23 13:12 wu-hanqing

cicheck

wu-hanqing avatar Dec 26 '23 02:12 wu-hanqing

cicheck

liuminjian avatar Dec 28 '23 07:12 liuminjian