blazingmq icon indicating copy to clipboard operation
blazingmq copied to clipboard

Performance[MQBSTAT]: lookup for per-appId metrics O(n) -> O(1)

Open 678098 opened this issue 1 year ago • 2 comments

Previously, we used a bsl::list<ManagedPtr<mwcst::StatContext> > for 2 roles:

  • Store managed pointers to subcontexts associated with appIds
  • Lookup to subcontexts for each specific appId

The problem is that the list assumes linear search, which might be too slow if we have a decent number of appIds.

However, due to the interfaces of ManagedPtr (on Solaris), we cannot just use a container which might reallocate its elements on resize. So instead we keep&update 2 collections together: bsl::list to hold ManagedPointers and bsl::unordered_map for fast lookup to raw pointers to the needed StatContexts

Full list of changes:

  • mqbstat::QueueStatsDomain: hold managed pointers in bsl::list<StatSubContextMp> d_subContextsHolder, lookup to subcontexts with bsl::unordered_map<bsl::string, mwcst::StatContext*> d_subContextsLookup
  • mqbstat::QueueStatsDomain: pass allocator in constructor
  • mqbstat::QueueStatsDomain: improve logging when no subcontext found
  • mqbc::ClusterDataIdentity: add safety check to prevent possible nullptr dereference
  • mqbmock::Cluster: add safety checks to find cluster misconfiguration early
  • mqbstat_queuestats.t.cpp: add UT to check per-appId subcontexts and metrics

678098 avatar Aug 05 '24 21:08 678098

However, due to the interfaces of ManagedPtr (on Solaris), we cannot just use a container which might reallocate its elements on resize.

Do you have source to this statement? It is quite inconvenient to have to use two data structures.

kaikulimu avatar Aug 09 '24 19:08 kaikulimu

Do you have source to this statement? It is quite inconvenient to have to use two data structures.

@kaikulimu ManagedPtr docs https://bloomberg.github.io/bde-resources/doxygen/bde_api_prod/bslma__managedptr_8h_source.html

However, like 'bsl::auto_ptr' it has unusual "copy-semantics" that transfer ownership of the managed object, rather than making a copy. It should be noted that this signature does not satisfy the requirements for an element-type stored in any of the standard library containers.

So ManagedPtr doesn't satisfy the CopyConstructible requirements https://en.cppreference.com/w/cpp/named_req/CopyConstructible

On SunOS, trying to build a bsl::unordered_map<bsl::string, bslma::ManagedPtr<...> > will cause compilation errors, so as with other container types which rely on reallocating its elements.

2024-08-07 13:57:33 | "/opt/bb/include/bslstl_pair.h", line 2528: Error: Could not find a match for BloombergLP::bslma::ManagedPtr<BloombergLP::mwcst::StatContext>::ManagedPtr<MANAGED_TYPE>(const BloombergLP::bslma::ManagedPtr<BloombergLP::mwcst::StatContext>) needed in void bsl::Pair_Second<BloombergLP::bslma::ManagedPtr<BloombergLP::mwcst::StatContext>>::Pair_Second<BloombergLP::bslma::ManagedPtr<BloombergLP::mwcst::StatContext>>(const BloombergLP::bslma::ManagedPtr<BloombergLP::mwcst::StatContext>&, BloombergLP::bslma::Allocator*, bsl::integral_constant<int, 0>). 2024-08-07 13:57:33 | "/opt/bb/include/bslstl_pair.h", line 2792: Where: While instantiating "void bsl::Pair_Second<BloombergLP::bslma::ManagedPtr<BloombergLP::mwcst::StatContext>>::Pair_Second<BloombergLP::bslma::ManagedPtr<BloombergLP::mwcst::StatContext>>(const BloombergLP::bslma::ManagedPtr<BloombergLP::mwcst::StatContext>&, BloombergLP::bslma::Allocator*, bsl::integral_constant<int, 0>)". 2024-08-07 13:57:33 | "/opt/bb/include/bslstl_pair.h", line 2792: Where: Instantiated from bsl::pair<const bsl::basic_string<char, std::char_traits<char>, bsl::allocator<char>>, BloombergLP::bslma::ManagedPtr<BloombergLP::mwcst::StatContext>>::pair(const bsl::basic_string<char, std::char_traits<char>, bsl::allocator<char>>&, const BloombergLP::bslma::ManagedPtr<BloombergLP::mwcst::StatContext>&, BloombergLP::bslma::Allocator*). 2024-08-07 13:57:33 | "/opt/bb/include/bslma_constructionutil_cpp03.h", line 9370: Where: Instantiated from static void bsl::allocator_traits<bsl::allocator<long double>>::construct<bsl::pair<const bsl::basic_string<char, std::char_traits<char>, bsl::allocator<char>>,

The first message could be simplified: Error: Could not find a match for ManagedPtr<StatContext>::ManagedPtr<MANAGED_TYPE>(const ManagedPtr<StatContext>)

If you check the ManagedPtr interfaces, there are no constructors accepting other ManagedPtrs qualified as const https://bloomberg.github.io/bde-resources/doxygen/bde_api_prod/classbslma_1_1ManagedPtr.html

678098 avatar Aug 12 '24 19:08 678098