apache
HIVE-26437: Dump unpartitioned tables in parallel
HIVE-26437: Dump unpartitioned tables in parallel.
Currently partitions of table is dump in parallel manner. But if table is not partitioned, it is dumped serially. This change introduces parallelism at table level as well. A single thread pool which is currently being used for partition level is also used for table level. The table level dump task is added to same thread pool. The degree of parallelism depends upon config parameter REPL_PARTITIONS_DUMP_PARALLELISM whose defaul value is 100. The new ExportService is introduced with this change which would be responsible for exporting table and partitions during repl dump. The ExportService is initialized and configured with thread pools by HiveServer2 service. A new Hiveconfig variable ie "REPL_TABLE_DUMP_PARALLELISM is introduced to define the number of threads which would be created in thread pool. The ExportService which is created as singleton instance would be used by ReplDumpTask.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?
Coding standard is 2 space indents.
The unit test mainly tests the DumpExporter since the runnable does nothing. Maybe there needs to be an integration test that makes sure that running exports in parallel doesn't have issues like more than one thread accessing some resource that we don't know about.
For your existing test, maybe the multithreaded one can do a runnable like this:
- counter that is a semaphore set to the number of currency that you want
- each runnable, counter.tryAcquire() - should not block (return false) sleep counter.release()
Need to test the test too.
Add a test like this:
public void testDumpExporterWithParallelismFail() throws Exception { when(conf.getIntVar(HiveConf.ConfVars.REPL_EXPORT_DUMP_PARALLELISM)).thenReturn(10); sem = new Semaphore(totalTask-1);
This test should fail intentionally. So you will need to catch the exception and PASS if it fails. I think there is an annotation for that like
@test(expected=NullPointerException.class)
replace with your exception
What is the failure expected? We have for e.g 7 semaphore limits and executing 8 runnable task. The 8th one wont get semaphore and return without incrementing task number. The ASSURE_TRUE(7,8) will fail. Is this the one you are saying?
Ignore these comments, I added them in line instead.
@test(expected = HiveException.class) public void testDumpExporterWithParallelismFail() throws Exception {
Shouldn't the error be AssertionError? Is what's failing really what you expected?
I think this is what's wrong:
Assert.assertTrue(sem.tryAcquire());
Should be different for positive vs negative test.
Positive tests should be true, Negative tests should be false.
Are all your other tests failing too? they may, you are catching and not rethrowing there.
This should be what's failing. assertTrue() throw
For performance benchmark, please see https://github.com/apache/hive/blob/master/itests/hive-jmh/src/main/java/org/apache/hive/benchmark/vectorization/VectorizedArithmeticBench.java . It's a JMH benchmark. It's simple and has good explanation on how to use it.
Results of jmh performance benchmark test indicates an improvement in TableExport operation. If table is dumped in parallel manner instead of serial , the operation completes much faster giving. Below is the result seen when 500 tablexport operations are done both in serial and parallel manner.
Result "org.apache.hive.benchmark.ql.exec.TableAndPartitionExportBench.BaseBench.parallel": N = 5 mean = 640.862 ?(99.9%) 113.354 ms/op
Result "org.apache.hive.benchmark.ql.exec.TableAndPartitionExportBench.BaseBench.serial": N = 5 mean = 51697.322 ?(99.9%) 322.747 ms/op
Benchmark Mode Cnt Score Error Units TableAndPartitionExportBench.BaseBench.parallel ss 5 640.862 ? 113.354 ms/op TableAndPartitionExportBench.BaseBench.serial ss 5 51697.322 ? 322.747 ms/op
The test failed. Please check http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3444/19/tests .