ozone
ozone copied to clipboard
apache
HDDS-10568. When the ldb command is executed, it is output by line
What changes were proposed in this pull request?
When executing the ldb command, if the data is very large, a very large file will be generated, which is not friendly. This pr will add a new function that can control the maximum number of records allowed to be saved in each file. Details: HDDS-10568
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10568
How was this patch tested?
The new command can be combined with --pre-file-records, using:
./bin/ozone debug ldb --db=/xxxx/xxxx/xxxx/om.db scan --column_family=fileTable --out=/xxxx/xxxx/xxxx/om/data/tmp_fileTable --pre-file-records=1000000
Effective file result set:
ci : https://github.com/jianghuazhu/ozone/actions/runs/8387158994
Can you help review this pr, @adoroszlai @xichen01 . Thanks.
Sorry, I had some other work some time ago. I will make up for this issue in the near future.
The unit tests run fine on my local machine:
Can you help me look at this PR again, @errose28 @xichen01 . Thanks.
https://github.com/apache/ozone/actions/runs/8849501009/job/24349282946?pr=6420 In CI, it is prompted that File-related operations cannot be performed, such as mkdirs(). Is it because it has something to do with our testing environment? @errose28 @kerneltime @adoroszlai , do you know the reason?
In CI, it is prompted that File-related operations cannot be performed, such as mkdirs().
M B RV: Exceptional return value of java.io.File.mkdirs() ignored in org.apache.hadoop.ozone.debug.DBScanner.displayTable(ManagedRocksIterator, DBColumnFamilyDefinition, boolean) At DBScanner.java:[line 232]
https://spotbugs.readthedocs.io/en/latest/bugDescriptions.html#rv-method-ignores-exceptional-return-value-rv-return-value-ignored-bad-practice
https://github.com/apache/ozone/blob/c7012f3bc91e92af08f2d89ee6ac1cbb103475d2/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/DBScanner.java#L228-L233
File.mkdirs() returns false if it could not create the directories. This return value needs to be handled (e.g. exit with error message).
M D NP: Possible null pointer dereference in org.apache.hadoop.ozone.debug.TestLDBCli.testScanWithRecordsPerFile() due to return value of called method Dereferenced at TestLDBCli.java:[line 286]
https://spotbugs.readthedocs.io/en/latest/bugDescriptions.html#np-possible-null-pointer-dereference-due-to-return-value-of-called-method-np-null-on-some-path-from-return-value
https://github.com/apache/ozone/blob/c7012f3bc91e92af08f2d89ee6ac1cbb103475d2/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/debug/TestLDBCli.java#L285-L286
File.listFiles returns null if the path is not a directory. Try checking if tmpDir1 is a directory. The same applies to tmpDir2 below.
ci: https://github.com/jianghuazhu/ozone/actions/runs/8876637070 I've updated it. Can you help me look at this PR again, @errose28 @xichen01 @adoroszlai ? Thanks.