ozone icon indicating copy to clipboard operation
ozone copied to clipboard

HDDS-10813. Improve Recon Debuggability for Snapshot Fetching, Sync Monitoring, and Permission Validation.

Open ArafatKhan2198 opened this issue 9 months ago • 1 comments

What changes were proposed in this pull request?

Changes in this patch include the following :-

Enhanced Logging for OM Snapshot Fetching

We included checks for the presence of SST files, logging a warning if the snapshot is empty or lacks SST files, which could indicate issues such as OM leader instability or snapshot corruption.

Logging Last Sequence Number on Periodic Syncs

The synchronization process between Recon and the OM database now includes logging of the last sequence number before and after each sync.

Permission Checks for Recon DB Directories

We implemented a method to validate the permissions of the Recon DB directories against the expected configuration settings. The method logs a warning if the actual permissions do not match the expected permissions, enhancing operational awareness.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10813

How was this patch tested?

Manually Tested out the logs. The required log messages get printed :-

  1. Logging for SST Files Availability on Full DB Fetch:
2024-05-06 02:13:03 2024-05-05 20:43:03,285 [Recon-SyncOM-0] INFO impl.OzoneManagerServiceProviderImpl: Valid OM snapshot with SST files found at: /data/metadata/om.snapshot.db_1714941781590
  1. Printing the Sequence Number Before and After Sync:
2024-05-06 02:13:01 2024-05-05 20:43:01,587 [Recon-SyncOM-0] INFO impl.OzoneManagerServiceProviderImpl: Last known sequence number before sync: 0
.....
2024-05-06 02:13:03 2024-05-05 20:43:03,620 [Recon-SyncOM-0] INFO impl.OzoneManagerServiceProviderImpl: Sequence number after sync: 2
  1. Recon Data Directory Permissions:
2024-05-06 02:13:01 2024-05-05 20:43:01,590 [Recon-SyncOM-0] INFO impl.OzoneManagerServiceProviderImpl: Permissions for Recon DB directory '/data/metadata' are correctly set to 'rwxr-x---'

ArafatKhan2198 avatar May 05 '24 20:05 ArafatKhan2198

@devmadhuu @dombizita Please take a look.

ArafatKhan2198 avatar May 05 '24 20:05 ArafatKhan2198

@devmadhuu Let me know if there is anything else remaining on this.

ArafatKhan2198 avatar Jun 13 '24 14:06 ArafatKhan2198

Thanks @sumitagrawl for review.

devmadhuu avatar Jun 25 '24 08:06 devmadhuu