Tendis
Tendis copied to clipboard
Published
20 hours ago •
Tencent
Tencent
cpu 还能降一降么?
Your Environment
- Operating System and version: centos 8.4 64位
- Machine Specifications: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz 56核, 160GB每台 nvme ssd 2TB 每台, 3台 3主3从
- Tendis Version: tendisplus-2.4.3-rocksdb-v5.13.4
- Tendis Configuration:
bind 0.0.0.0
port 2000
daemon yes
cluster-enabled yes
requirepass tendis99
masterauth tendis99
loglevel notice
logdir /data1/tendis/tendisplus/log/2000
dumpdir /data1/tendis/tendisplus/dump/2000
dir /data1/tendis/tendisplus/db/2000
pidfile /data1/tendis/tendisplus/tendisplus-2000.pid
slowlog /data1/tendis/tendisplus/log/2000/slowlog
rocks.blockcachemb 16384
rocks.compress_type none // 或者 lz4
executorThreadNum 48
netiothreadnum 12
Description
测试下来, snappy 和 none 压缩算法, cpu都要到2000% , lz4 在1800% 左右, 还有办法调优 降低一点么,
下面是perf top的截图
一主一从 LZ4
Samples: 4M of event 'cycles', 4000 Hz, Event count (approx.): 995508402247 lost: 0/0 drop: 0/419807
Overhead Shared Object Symbol
2.20% tendisplus [.] rocksdb::BlockIter::BinarySeek
1.84% tendisplus [.] rocksdb::(anonymous namespace)::FullFilterBitsReader::MayMatch
1.84% tendisplus [.] rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::FindGreaterOrEqual
1.79% libc-2.28.so [.] __memcmp_avx2_movbe
1.77% tendisplus [.] rocksdb::Version::Get
1.63% [kernel] [k] do_syscall_64
1.48% tendisplus [.] free
1.45% tendisplus [.] rocksdb::(anonymous namespace)::BytewiseComparatorImpl::Compare
1.35% tendisplus [.] rocksdb::InternalKeyComparator::Compare
1.23% tendisplus [.] rocksdb::InternalKeyComparator::CompareKeySeq
1.11% tendisplus [.] malloc
1.10% tendisplus [.] rocksdb::BlockIter::ParseNextKey
1.09% tendisplus [.] LZ4_decompress_safe
1.08% libpthread-2.28.so [.] __pthread_mutex_lock
1.07% libc-2.28.so [.] __memmove_avx_unaligned_erms
1.02% tendisplus [.] rocksdb::MemTable::KeyComparator::operator()
0.75% libpthread-2.28.so [.] __pthread_mutex_unlock_usercnt
0.71% tendisplus [.] tendisplus::NetSession::setResponse
0.65% tendisplus [.] rocksdb::Hash
0.64% libc-2.28.so [.] vfprintf
0.62% tendisplus [.] rocksdb::BlockBasedTable::Get
0.62% [kernel] [k] switch_mm_irqs_off
0.62% [kernel] [k] syscall_return_via_sysret
0.60% tendisplus [.] rocksdb::TableCache::Get
0.57% [kernel] [k] __schedule
0.56% tendisplus [.] rocksdb::Block::NewIterator
0.54% [kernel] [k] entry_SYSCALL_64
0.51% libc-2.28.so [.] sched_getcpu
0.50% tendisplus [.] rocksdb::StatisticsImpl::recordTick
0.50% [kernel] [k] copy_user_enhanced_fast_string
0.48% [kernel] [k] menu_select
0.47% tendisplus [.] rocksdb::DBImpl::GetImpl
0.46% tendisplus [.] std::vector<unsigned char, std::allocator<unsigned char> >::_M_range_insert<__gnu_cxx::__normal_iterator<char const*
0.44% [kernel] [k] _raw_spin_lock_irqsave
0.41% tendisplus [.] rocksdb::LRUHandleTable::FindPointer
0.39% [kernel] [k] _raw_spin_lock
0.38% tendisplus [.] rocksdb::HistogramBucketMapper::IndexForValue
0.35% [kernel] [k] futex_wake
Too slow to read ring buffer (change period (-c/-F) or limit CPUs (-C)
一主一从 none 压缩
Samples: 894K of event 'cycles', 4000 Hz, Event count (approx.): 401056174274 lost: 0/0 drop: 0/459196
Overhead Shared Object Symbol
2.79% tendisplus [.] snappy::RawUncompress
2.45% tendisplus [.] rocksdb::BlockIter::BinarySeek
2.08% tendisplus [.] rocksdb::(anonymous namespace)::FullFilterBitsReader::MayMatch
2.00% tendisplus [.] rocksdb::Version::Get
1.90% libc-2.28.so [.] __memcmp_avx2_movbe
1.87% tendisplus [.] rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::FindGreaterOrEqual
1.59% [kernel] [k] do_syscall_64
1.55% tendisplus [.] rocksdb::(anonymous namespace)::BytewiseComparatorImpl::Compare
1.54% tendisplus [.] free
1.45% tendisplus [.] rocksdb::InternalKeyComparator::Compare
1.29% tendisplus [.] rocksdb::InternalKeyComparator::CompareKeySeq
1.20% tendisplus [.] malloc
1.17% tendisplus [.] rocksdb::BlockIter::ParseNextKey
1.15% libpthread-2.28.so [.] __pthread_mutex_lock
1.07% tendisplus [.] rocksdb::MemTable::KeyComparator::operator()
1.06% libc-2.28.so [.] __memmove_avx_unaligned_erms
0.81% libpthread-2.28.so [.] __pthread_mutex_unlock_usercnt
0.77% tendisplus [.] rocksdb::TableCache::Get
0.76% tendisplus [.] tendisplus::NetSession::setResponse
0.70% tendisplus [.] rocksdb::BlockBasedTable::Get
0.68% tendisplus [.] rocksdb::Hash
0.64% libc-2.28.so [.] vfprintf
0.62% [kernel] [k] syscall_return_via_sysret
0.58% [kernel] [k] switch_mm_irqs_off
0.57% libc-2.28.so [.] sched_getcpu
0.56% [kernel] [k] __schedule
0.56% tendisplus [.] rocksdb::StatisticsImpl::recordTick
0.55% tendisplus [.] rocksdb::Block::NewIterator
0.54% [kernel] [k] copy_user_enhanced_fast_string
0.53% [kernel] [k] update_load_avg
0.52% [kernel] [k] entry_SYSCALL_64
0.49% [kernel] [k] menu_select
0.47% tendisplus [.] rocksdb::DBImpl::GetImpl
0.47% tendisplus [.] rocksdb::LRUHandleTable::FindPointer
0.46% tendisplus [.] rocksdb::HistogramBucketMapper::IndexForValue
0.44% [kernel] [k] _raw_spin_lock_irqsave
0.42% tendisplus [.] std::vector<unsigned char, std::allocator<unsigned char> >::_M_range_insert<__gnu_cxx::__normal_iterator<char const*,
0.42% [vdso] [.] __vdso_getcpu
Too slow to read ring buffer (change period (-c/-F) or limit CPUs (-C)
最后一主一从 lz4
Samples: 856K of event 'cycles', 4000 Hz, Event count (approx.): 372364926233 lost: 0/0 drop: 0/458227
Overhead Shared Object Symbol
2.61% tendisplus [.] rocksdb::BlockIter::BinarySeek
2.22% tendisplus [.] rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::FindGreaterOrEqual
2.02% tendisplus [.] rocksdb::Version::Get
2.01% libc-2.28.so [.] __memcmp_avx2_movbe
1.86% tendisplus [.] LZ4_decompress_safe
1.84% tendisplus [.] rocksdb::(anonymous namespace)::FullFilterBitsReader::MayMatch
1.68% [kernel] [k] do_syscall_64
1.65% tendisplus [.] rocksdb::(anonymous namespace)::BytewiseComparatorImpl::Compare
1.56% tendisplus [.] rocksdb::InternalKeyComparator::Compare
1.48% tendisplus [.] free
1.41% tendisplus [.] rocksdb::InternalKeyComparator::CompareKeySeq
1.21% libpthread-2.28.so [.] __pthread_mutex_lock
1.21% tendisplus [.] rocksdb::BlockIter::ParseNextKey
1.20% tendisplus [.] rocksdb::MemTable::KeyComparator::operator()
1.18% tendisplus [.] malloc
1.15% libjvm.so [.] SpinPause
0.90% libc-2.28.so [.] __memmove_avx_unaligned_erms
0.84% libpthread-2.28.so [.] __pthread_mutex_unlock_usercnt
0.83% tendisplus [.] tendisplus::NetSession::setResponse
0.70% tendisplus [.] rocksdb::BlockBasedTable::Get
0.69% tendisplus [.] rocksdb::TableCache::Get
0.65% [kernel] [k] switch_mm_irqs_off
0.64% [kernel] [k] syscall_return_via_sysret
0.64% tendisplus [.] rocksdb::Block::NewIterator
0.63% [kernel] [k] copy_user_enhanced_fast_string
0.63% libc-2.28.so [.] vfprintf
0.60% tendisplus [.] rocksdb::Hash
0.59% libc-2.28.so [.] sched_getcpu
0.56% tendisplus [.] rocksdb::StatisticsImpl::recordTick
0.55% [kernel] [k] entry_SYSCALL_64
0.54% [kernel] [k] __schedule
0.52% tendisplus [.] rocksdb::DBImpl::GetImpl
0.50% [kernel] [k] menu_select
0.50% libc-2.28.so [.] __memmove_sse2_unaligned_erms
0.50% liblz4-java3672842715170129947.so [.] LZ4_decompress_fast
0.47% tendisplus [.] rocksdb::LRUHandleTable::FindPointer
0.46% tendisplus [.] rocksdb::HistogramBucketMapper::IndexForValue
0.45% [kernel] [k] _raw_spin_lock_irqsave
Too slow to read ring buffer (change period (-c/-F) or limit CPUs (-C)
另外 就是 配置的16GB 实际占用的是22GB, 6个节点就是120GB左右了 总共
这个cpu使用率2000%是正常的哦，关键还是看你压测的qps有多大。一般qps到了20万级别的话，cpu利用率就很容易到2000%的。 内存使用方面，参考我在这个issue里面的回复：https://github.com/Tencent/Tendis/issues/167 https://github.com/Tencent/Tendis/issues/167#issuecomment-1192148757
