v6d icon indicating copy to clipboard operation
v6d copied to clipboard

Revise perfect hash to align with libgrape-lite's pthash

Open vegetableysm opened this issue 1 year ago • 1 comments

What do these changes do?

OLD: Modifications of libgrape-lite perfect hash

  • remove inline int8_t log2(size_t value) function from hashmap_indexer_impl.h(Not used)
  • remove namespace sync_comm and struct CommImpl from hashmap_indexer_impl.h(Not used)
  • remove Allocator<T> of std::vector<T, Allocator<T>> inner_ in hashmap_indexer_impl.h(No need)
  • remove encode_vec encode_val and decode_val API from ref_vector.h (Not used)
  • add struct murmurhasher to single_phf_view.h (reduce dependency)
  • add struct external_mem_dumper to single_phf_view.h for dump data to blob (Reduce memcpy)
  • remove std::set<size_t> idx in static void build(Iterator keys, uint64_t n, Dumper& dumper, int thread_num) of struct SinglePHFView in single_phf_view.h (No need)
  • add static void build(Iterator keys, uint64_t n, pthash::single_phf<murmurhasher, pthash::dictionary_dictionary, true>& phf, int thread_num) API for struct SinglePHFView in single_phf_view.h (Reduce memcpy)
  • remove void serialize(std::unique_ptr<IOADAPTOR_T>& writer) void deserialize(std::unique_ptr<IOADAPTOR_T>& reader) and void serialize_to_mem(std::vector<char>& buf) from class StringViewVector in string_view_vector.h (Not used)
  • remove void Serialize(const std::string& path) and void Deserialize(const std::string& path) from class ImmPHIdxer in perfect_hash_indexer.h (Not used)
  • overwrite finish() API of class PHIdxerViewBuilder in perfect_hash_indexer.h (For vineyard)
  • replace std::vector<char> buffer_ with std::shared_ptr<Blob> buffer_ in perfect_hash_indexer.h (For vineyard)
  • replace nonstd::string_view with arrow_string_view in all files. (For vineyard)

NEW:

  • Use libgrapelite pthash
  • Remove pthash / BBHash from vineyard

Related issue number

Fixes #1852

vegetableysm avatar Mar 29 '24 07:03 vegetableysm

/cc @sighingnow, this issus/pr has had no activity for a long time, please help to review the status and assign people to work on it.

github-actions[bot] avatar May 12 '24 00:05 github-actions[bot]

Refer to #1992

vegetableysm avatar Aug 23 '24 09:08 vegetableysm