linkedin
[venice-server] Dropping unassigned partitions
Summary, imperative, start upper case, don't end with a period
The goal is to remove unassigned partitions that are on disk state
This involves storage service and engine, which upon starting (constructed), will go through folders on disk and verify status of folders Helix's ideal state is used for consulting partitions and their assignment The change proposed includes defining a function in VeniceServer that goes through partitions in idealState and compares to partitions in storageEngine StorageEngine object initialization is also being updated to run the new function that performs this verification.
Resolves #650
How was this PR tested?
Will write corresponding test.
Does this PR introduce any user-facing changes?
- [X] No. You can skip the rest of this section.
- [ ] Yes. Make sure to explain your proposed changes and call out the behavior change.
It looks like spotBugs is failing.
If you run this locally, you'll be able to read the report to see what's wrong.
./gradlew :services:venice-server:spotbugsMain
Make sure to build before running this.
A few tests still have yet to pass with the changes that include the check for partition assignment. [e.g. this test] I have to check whether these are related to the changes I've submitted; also some of these tests seem to pass locally.
(Suggested addition to code, that seems directly useful but is no longer / not used actively:
In the HelixParticipationService file, similar to the getInstance function, I also wrote a getHelixManager function. I tried using this before in access after HelixParticipationService is instantiated, but have changed this implementation. )
(Suggested addition to code, that seems directly useful but is no longer / not used actively:
In the
HelixParticipationServicefile, similar to the getInstance function, I also wrote a getHelixManager function. I tried using this before in access afterHelixParticipationServiceis instantiated, but have changed this implementation. )
(Reverted code in this file back to original state but thought I should make a note of this).
I wrote the related unit test for this PR. GitHub tests pass for program, except for one Integration test unrelated to my change.
The check over which storage partitions to persist is moved to the shutdown phase of VeniceServer which was discussed.
Shutdown phase contains call to storageService.checkWhetherStoragePartitionsShouldBeKeptOrNot. Services are also stopped when VeniceServer shuts down.
Congratulations on merging your first PR into Venice @kristyelee! Thank you for your persistence. 🚀
[celebrate] Manoj Nagarajan reacted to your message:
From: Koorous Vargha @.> Sent: Tuesday, November 19, 2024 10:08:54 PM To: linkedin/venice @.> Cc: Subscribed @.***> Subject: Re: [linkedin/venice] [server] Dropping unassigned partitions (PR #1196)
Congratulations on merging your first PR into Venice @kristyeleehttps://github.com/kristyelee. Thank you for your persistence.
— Reply to this email directly, view it on GitHubhttps://github.com/linkedin/venice/pull/1196#issuecomment-2486852299, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADJ2CHLIYEPI6QMITO2OT532BOZGXAVCNFSM6AAAAABOXJAXF2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBWHA2TEMRZHE. You are receiving this because you are subscribed to this thread.Message ID: @.***>
