raw client batch put memory leak

[oswaldoooo]

valgrindbak.log this is my db bench valgrind analyze result. i need put 10 million custom record data to tikv for query loadtest.

And I've identified a memory leak issue in the client application. As the data volume increases, the memory usage grows progressively. When inserting 10 million records, the memory consumption reaches 700MB, which is unreasonable given that:

My concurrency is strictly limited to 100,000 operations and I only have about 60 bytes per record The memory usage accumulates gradually over time This linear growth pattern suggests unmanaged resource retention

:=/

[pingyu]

Thanks for you report !

I think the memory is mostly used by the region cache. Which is also show in the valgrind result:

==1551== 4 bytes in 1 blocks are still reachable in loss record 5 of 293
==1551==    at 0x4C39185: malloc (vg_replace_malloc.c:442)
==1551==    by 0x3CF985: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x3B00AE: prometheus::counter::<impl prometheus::vec::MetricVec<prometheus::counter::CounterVecBuilder<P>>>::new (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x1CB315: spin::once::Once<T,R>::try_call_once_slow (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x3FAF9B: <tikv_client::pd::retry::RetryClient as tikv_client::pd::retry::RetryClientTrait>::get_region::{{closure}} (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x306B26: <tikv_client::pd::client::PdRpcClient<Cod,KvC> as tikv_client::pd::client::PdClient>::region_for_key::{{closure}} (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x326F99: <tikv_client::compat::LoopFn<A,F> as futures_core::stream::Stream>::poll_next (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x2AAAC0: <futures_util::stream::try_stream::and_then::AndThen<St,Fut,F> as futures_core::stream::Stream>::poll_next (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x27672A: <futures_util::stream::stream::collect::Collect<St,C> as core::future::future::Future>::poll (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x2A6744: tikv_client::request::plan::RetryableMultiRegion<P,PdC>::single_plan_handler::{{closure}} (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x29EE4B: <tikv_client::request::plan::RetryableMultiRegion<P,PdC> as tikv_client::request::plan::Plan>::execute::{{closure}} (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x29D3E0: <tikv_client::request::plan::ExtractError<P> as tikv_client::request::plan::Plan>::execute::{{closure}} (in /home/oswaldo/imgo/db-bench/target/release/db-bench)

Maybe we can try to reduce the memory consumption. What's the maximum number of regions during your bench ?

Besides, the valgrind result show the memory leak on static objects created by lazy_static. It would be an issue of lazy_build, or false positive report. I will check this later.

==1551== Thread 1:
==1551== 4 bytes in 1 blocks are still reachable in loss record 1 of 293
==1551==    at 0x4C39185: malloc (vg_replace_malloc.c:442)
==1551==    by 0x55106A: regex_automata::nfa::thompson::builder::Builder::build (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x5492F7: regex_automata::nfa::thompson::compiler::Compiler::compile (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x522DB6: regex_automata::meta::strategy::new (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x55645A: regex_automata::meta::regex::Builder::build (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x51FDF2: regex::builders::string::RegexBuilder::build (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x51F61C: regex::regex::string::Regex::new (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x1CB052: spin::once::Once<T,R>::try_call_once_slow (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x3C503F: <tikv_client::common::security::SCHEME_REG as core::ops::deref::Deref>::deref (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x29F3DF: tikv_client::pd::cluster::Connection::connect::{{closure}} (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x2B9399: <db_bench::database::tikv::Database as db_bench::config::Initer>::new::{{closure}} (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x2029C1: tokio::runtime::park::CachedParkThread::block_on (in /home/oswaldo/imgo/db-bench/target/release/db-bench)

==1551== 4 bytes in 1 blocks are still reachable in loss record 3 of 293
==1551==    at 0x4C39185: malloc (vg_replace_malloc.c:442)
==1551==    by 0x485985: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x48B483: prometheus::histogram::<impl prometheus::vec::MetricVec<prometheus::histogram::HistogramVecBuilder>>::new (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x1CC0A4: spin::once::Once<T,R>::try_call_once_slow (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x3FAF84: <tikv_client::pd::retry::RetryClient as tikv_client::pd::retry::RetryClientTrait>::get_region::{{closure}} (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x306B26: <tikv_client::pd::client::PdRpcClient<Cod,KvC> as tikv_client::pd::client::PdClient>::region_for_key::{{closure}} (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x326F99: <tikv_client::compat::LoopFn<A,F> as futures_core::stream::Stream>::poll_next (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x2AAAC0: <futures_util::stream::try_stream::and_then::AndThen<St,Fut,F> as futures_core::stream::Stream>::poll_next (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x27672A: <futures_util::stream::stream::collect::Collect<St,C> as core::future::future::Future>::poll (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x2A6744: tikv_client::request::plan::RetryableMultiRegion<P,PdC>::single_plan_handler::{{closure}} (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x29EE4B: <tikv_client::request::plan::RetryableMultiRegion<P,PdC> as tikv_client::request::plan::Plan>::execute::{{closure}} (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x29D3E0: <tikv_client::request::plan::ExtractError<P> as tikv_client::request::plan::Plan>::execute::{{closure}} (in /home/oswaldo/imgo/db-bench/target/release/db-bench)

==1551== 4 bytes in 1 blocks are still reachable in loss record 11 of 293
==1551==    at 0x4C39185: malloc (vg_replace_malloc.c:442)
==1551==    by 0x485985: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x48B483: prometheus::histogram::<impl prometheus::vec::MetricVec<prometheus::histogram::HistogramVecBuilder>>::new (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x1CC424: spin::once::Once<T,R>::try_call_once_slow (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x479DF6: tikv_client::stats::tikv_stats (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x29C996: <tikv_client::request::plan::Dispatch<Req> as tikv_client::request::plan::Plan>::execute::{{closure}} (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x2A7B85: tikv_client::request::plan::RetryableMultiRegion<P,PdC>::single_shard_handler::{{closure}} (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x33605F: tokio::runtime::task::core::Core<T,S>::poll (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x35A356: tokio::runtime::task::harness::Harness<T,S>::poll (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x7D495A: tokio::runtime::scheduler::multi_thread::worker::Context::run_task (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x7D392A: tokio::runtime::scheduler::multi_thread::worker::Context::run (in /home/oswaldo/imgo/db-bench/target/release/db-bench)
==1551==    by 0x7D78A6: tokio::runtime::context::scoped::Scoped<T>::set (in /home/oswaldo/imgo/db-bench/target/release/db-bench)