iotdb icon indicating copy to clipboard operation
iotdb copied to clipboard
verified publisher icon apache

[IOTDB-2569] Support ZSTD Compression use airlift

Open lyssom opened this issue 3 years ago • 11 comments

Description

Support ZSTD Compression use airlift

zstd use JNI and performance test please see: https://github.com/apache/iotdb/pull/6599

Content1 ...

Content2 ...

Content3 ...

This PR has:

  • [ ] been self-reviewed.
    • [ ] concurrent read
    • [ ] concurrent write
    • [ ] concurrent read and write
  • [ ] added documentation for new or modified features or behaviors.
  • [ ] added Javadocs for most classes and all non-trivial methods.
  • [ ] added or updated version, license, or notice information
  • [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage.
  • [ ] added integration tests.
  • [ ] been tested in a test IoTDB cluster.
Key changed/added classes (or packages if there are too many classes) in this PR

lyssom avatar Jul 20 '22 06:07 lyssom

55901658396810_ pic

HTHou avatar Jul 21 '22 09:07 HTHou

8C32G datanode/conf/datanode-env.sh MAX_HEAP_SIZE="16G"

datanode/conf/iotdb-datanode.properties wal_buffer_size_in_byte=1048576 default_storage_group_level=2 compressor=ZSTD config.properties.txt

HTHou avatar Jul 21 '22 09:07 HTHou

55901658396810_ pic

sorry, It's airlift bug in ver 0.20 https://github.com/airlift/aircompressor/issues/122 and It's fix in ver0.21 https://github.com/airlift/aircompressor/pull/135

I will change version

lyssom avatar Jul 22 '22 10:07 lyssom

Hi, would you mind doing some performance tests?

HTHou avatar Jul 26 '22 09:07 HTHou

sure, please do performance tests, THANKS

lyssom avatar Jul 27 '22 10:07 lyssom

Actually... What I mean is do you have some tests result of this PR? 😆

HTHou avatar Jul 28 '22 06:07 HTHou

SNAPPY

2022-07-28 18:26:02,294 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.e.f.MemTableFlushTask:95 - The memTable size of SG root.test-0 is 1124046000, the avg series points num in chunk is 2310, total timeseries number is 25000 
2022-07-28 18:26:04,285 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.e.f.MemTableFlushTask:177 - Storage group root.test-0 memtable PrimitiveMemTable{planIndex=[0,0]} flushing a memtable has finished! Time consumption: 1991ms 
2022-07-28 18:26:04,615 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.c.a.CompressionRatio:98 - Compression ratio is 58.469725192679576 
2022-07-28 18:26:04,733 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.e.s.TsFileProcessor:1424 - Storage group root.test-0 close the file /data2/data/sequence/root.test/0/0/1659003958469-33-0-0.tsfile, TsFile size is 20195302, time consumption of flushing metadata is 117ms

ZSTD

2022-07-28 17:31:40,842 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.e.f.MemTableFlushTask:95 - The memTable size of SG root.test-0 is 1117233600, the avg series points num in chunk is 2296, total timeseries number is 25000 
2022-07-28 17:31:44,075 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.e.f.MemTableFlushTask:177 - Storage group root.test-0 memtable PrimitiveMemTable{planIndex=[0,0]} flushing a memtable has finished! Time consumption: 3232ms 
2022-07-28 17:31:44,438 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.c.a.CompressionRatio:98 - Compression ratio is 98.23160330411592 
2022-07-28 17:31:44,539 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.e.s.TsFileProcessor:1424 - Storage group root.test-0 close the file /data2/data/sequence/root.test/0/0/1659000697005-44-0-0.tsfile, TsFile size is 12651862, time consumption of flushing metadata is 101ms

HTHou avatar Jul 28 '22 10:07 HTHou

Actually... What I mean is do you have some tests result of this PR? 😆

I'm sorry, I misunderstood.😂

lyssom avatar Jul 29 '22 02:07 lyssom

SNAPPY

2022-07-28 18:26:02,294 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.e.f.MemTableFlushTask:95 - The memTable size of SG root.test-0 is 1124046000, the avg series points num in chunk is 2310, total timeseries number is 25000 
2022-07-28 18:26:04,285 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.e.f.MemTableFlushTask:177 - Storage group root.test-0 memtable PrimitiveMemTable{planIndex=[0,0]} flushing a memtable has finished! Time consumption: 1991ms 
2022-07-28 18:26:04,615 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.c.a.CompressionRatio:98 - Compression ratio is 58.469725192679576 
2022-07-28 18:26:04,733 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.e.s.TsFileProcessor:1424 - Storage group root.test-0 close the file /data2/data/sequence/root.test/0/0/1659003958469-33-0-0.tsfile, TsFile size is 20195302, time consumption of flushing metadata is 117ms

ZSTD

2022-07-28 17:31:40,842 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.e.f.MemTableFlushTask:95 - The memTable size of SG root.test-0 is 1117233600, the avg series points num in chunk is 2296, total timeseries number is 25000 
2022-07-28 17:31:44,075 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.e.f.MemTableFlushTask:177 - Storage group root.test-0 memtable PrimitiveMemTable{planIndex=[0,0]} flushing a memtable has finished! Time consumption: 3232ms 
2022-07-28 17:31:44,438 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.c.a.CompressionRatio:98 - Compression ratio is 98.23160330411592 
2022-07-28 17:31:44,539 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.e.s.TsFileProcessor:1424 - Storage group root.test-0 close the file /data2/data/sequence/root.test/0/0/1659000697005-44-0-0.tsfile, TsFile size is 12651862, time consumption of flushing metadata is 101ms

ok, thank you. Thanks for tell me what can I do next?

lyssom avatar Aug 01 '22 02:08 lyssom

SNAPPY

2022-07-28 18:26:02,294 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.e.f.MemTableFlushTask:95 - The memTable size of SG root.test-0 is 1124046000, the avg series points num in chunk is 2310, total timeseries number is 25000 
2022-07-28 18:26:04,285 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.e.f.MemTableFlushTask:177 - Storage group root.test-0 memtable PrimitiveMemTable{planIndex=[0,0]} flushing a memtable has finished! Time consumption: 1991ms 
2022-07-28 18:26:04,615 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.c.a.CompressionRatio:98 - Compression ratio is 58.469725192679576 
2022-07-28 18:26:04,733 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.e.s.TsFileProcessor:1424 - Storage group root.test-0 close the file /data2/data/sequence/root.test/0/0/1659003958469-33-0-0.tsfile, TsFile size is 20195302, time consumption of flushing metadata is 117ms

ZSTD

2022-07-28 17:31:40,842 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.e.f.MemTableFlushTask:95 - The memTable size of SG root.test-0 is 1117233600, the avg series points num in chunk is 2296, total timeseries number is 25000 
2022-07-28 17:31:44,075 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.e.f.MemTableFlushTask:177 - Storage group root.test-0 memtable PrimitiveMemTable{planIndex=[0,0]} flushing a memtable has finished! Time consumption: 3232ms 
2022-07-28 17:31:44,438 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.c.a.CompressionRatio:98 - Compression ratio is 98.23160330411592 
2022-07-28 17:31:44,539 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.e.s.TsFileProcessor:1424 - Storage group root.test-0 close the file /data2/data/sequence/root.test/0/0/1659000697005-44-0-0.tsfile, TsFile size is 12651862, time consumption of flushing metadata is 101ms

ok, thank you. Thanks for tell me what can I do next?

Any idea about how to reduce the encoding time? The current version takes more time than SNAPPY.

HTHou avatar Aug 01 '22 07:08 HTHou

SNAPPY

2022-07-28 18:26:02,294 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.e.f.MemTableFlushTask:95 - The memTable size of SG root.test-0 is 1124046000, the avg series points num in chunk is 2310, total timeseries number is 25000 
2022-07-28 18:26:04,285 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.e.f.MemTableFlushTask:177 - Storage group root.test-0 memtable PrimitiveMemTable{planIndex=[0,0]} flushing a memtable has finished! Time consumption: 1991ms 
2022-07-28 18:26:04,615 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.c.a.CompressionRatio:98 - Compression ratio is 58.469725192679576 
2022-07-28 18:26:04,733 [pool-6-IoTDB-Flush-1] INFO  o.a.i.d.e.s.TsFileProcessor:1424 - Storage group root.test-0 close the file /data2/data/sequence/root.test/0/0/1659003958469-33-0-0.tsfile, TsFile size is 20195302, time consumption of flushing metadata is 117ms

ZSTD

2022-07-28 17:31:40,842 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.e.f.MemTableFlushTask:95 - The memTable size of SG root.test-0 is 1117233600, the avg series points num in chunk is 2296, total timeseries number is 25000 
2022-07-28 17:31:44,075 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.e.f.MemTableFlushTask:177 - Storage group root.test-0 memtable PrimitiveMemTable{planIndex=[0,0]} flushing a memtable has finished! Time consumption: 3232ms 
2022-07-28 17:31:44,438 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.c.a.CompressionRatio:98 - Compression ratio is 98.23160330411592 
2022-07-28 17:31:44,539 [pool-6-IoTDB-Flush-12] INFO  o.a.i.d.e.s.TsFileProcessor:1424 - Storage group root.test-0 close the file /data2/data/sequence/root.test/0/0/1659000697005-44-0-0.tsfile, TsFile size is 12651862, time consumption of flushing metadata is 101ms

ok, thank you. Thanks for tell me what can I do next?

Any idea about how to reduce the encoding time? The current version takes more time than SNAPPY. I will explore

lyssom avatar Aug 02 '22 07:08 lyssom